[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:40:02 dist-test-slave-b28r krb5kdc[19284](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:40:02 dist-test-slave-b28r krb5kdc[19284](info): set up 2 sockets
Sep 28 06:40:02 dist-test-slave-b28r krb5kdc[19284](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 06:40:03.483913 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:40:03 dist-test-slave-b28r krb5kdc[19284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:40:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:40:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.211.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
I20260928 06:40:03.595739 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:41425
--webserver_interface=127.18.211.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:41971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:41425
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:03.674772 19297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:03.674971 19297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:03.674991 19297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:03.676131 19297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:40:03.676172 19297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:03.676185 19297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:03.676196 19297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:40:03.676208 19297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:40:03.677642 19297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:41971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:41425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:41425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:03.677882 19297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:03.678092 19297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:03.683172 19302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:03.683230 19297 server_base.cc:1034] running on GCE node
W20260928 06:40:03.683362 19303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:03.683236 19305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:03.683614 19297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:03.683864 19297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:03.685015 19297 hybrid_clock.cc:648] HybridClock initialized: now 1790577603685001 us; error 33 us; skew 500 ppm
Sep 28 06:40:03 dist-test-slave-b28r krb5kdc[19284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577603, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:03.693073 19297 init.cc:377] Logged in from keytab as kudu/127.18.211.126@KRBTEST.COM (short username kudu)
I20260928 06:40:03.693620 19297 webserver.cc:460] Webserver started at http://127.18.211.126:36933/ using document root <none> and password file <none>
I20260928 06:40:03.693899 19297 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:03.693960 19297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:03.694113 19297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:03.695382 19297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "a6737b05b4a2459cbacb954cab3e5a3a"
format_stamp: "Formatted at 2026-09-28 06:40:03 on dist-test-slave-b28r"
server_key: "344c760860b9667d23cd576e70ddf427"
server_key_iv: "3b09ccd3676965753c33e331d81b0de9"
server_key_version: "encryptionkey@0"
I20260928 06:40:03.695737 19297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "a6737b05b4a2459cbacb954cab3e5a3a"
format_stamp: "Formatted at 2026-09-28 06:40:03 on dist-test-slave-b28r"
server_key: "344c760860b9667d23cd576e70ddf427"
server_key_iv: "3b09ccd3676965753c33e331d81b0de9"
server_key_version: "encryptionkey@0"
I20260928 06:40:03.697289 19297 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:40:03.698164 19312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:03.698390 19297 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:03.698477 19297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "a6737b05b4a2459cbacb954cab3e5a3a"
format_stamp: "Formatted at 2026-09-28 06:40:03 on dist-test-slave-b28r"
server_key: "344c760860b9667d23cd576e70ddf427"
server_key_iv: "3b09ccd3676965753c33e331d81b0de9"
server_key_version: "encryptionkey@0"
I20260928 06:40:03.698553 19297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:03.710068 19297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:03.711966 19297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:03.712095 19297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:03.715837 19297 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:41425
I20260928 06:40:03.715873 19364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:41425 every 8 connection(s)
I20260928 06:40:03.716197 19297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:40:03.717017 19365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:03.719297 19365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6737b05b4a2459cbacb954cab3e5a3a: Bootstrap starting.
I20260928 06:40:03.719933 19365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6737b05b4a2459cbacb954cab3e5a3a: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:03.720177 19365 log.cc:824] T 00000000000000000000000000000000 P a6737b05b4a2459cbacb954cab3e5a3a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:03.720327 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 19297
I20260928 06:40:03.720434 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:40:03.720871 19365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6737b05b4a2459cbacb954cab3e5a3a: No bootstrap required, opened a new log
I20260928 06:40:03.721078 19277 external_mini_cluster.cc:1428] Setting key 1e665c224a934c5709e77d445af7de0d
I20260928 06:40:03.722994 19365 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6737b05b4a2459cbacb954cab3e5a3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6737b05b4a2459cbacb954cab3e5a3a" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41425 } }
I20260928 06:40:03.723253 19365 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6737b05b4a2459cbacb954cab3e5a3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:03.723302 19365 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6737b05b4a2459cbacb954cab3e5a3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6737b05b4a2459cbacb954cab3e5a3a, State: Initialized, Role: FOLLOWER
I20260928 06:40:03.723435 19365 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6737b05b4a2459cbacb954cab3e5a3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6737b05b4a2459cbacb954cab3e5a3a" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41425 } }
I20260928 06:40:03.723520 19365 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6737b05b4a2459cbacb954cab3e5a3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:40:03.723559 19365 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6737b05b4a2459cbacb954cab3e5a3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:40:03.723609 19365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6737b05b4a2459cbacb954cab3e5a3a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:03.724697 19365 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6737b05b4a2459cbacb954cab3e5a3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6737b05b4a2459cbacb954cab3e5a3a" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41425 } }
I20260928 06:40:03.724814 19365 leader_election.cc:302] T 00000000000000000000000000000000 P a6737b05b4a2459cbacb954cab3e5a3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6737b05b4a2459cbacb954cab3e5a3a; no voters:
I20260928 06:40:03.724970 19365 leader_election.cc:288] T 00000000000000000000000000000000 P a6737b05b4a2459cbacb954cab3e5a3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:40:03.724990 19370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6737b05b4a2459cbacb954cab3e5a3a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:03.725102 19370 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6737b05b4a2459cbacb954cab3e5a3a [term 1 LEADER]: Becoming Leader. State: Replica: a6737b05b4a2459cbacb954cab3e5a3a, State: Running, Role: LEADER
I20260928 06:40:03.725183 19370 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6737b05b4a2459cbacb954cab3e5a3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6737b05b4a2459cbacb954cab3e5a3a" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41425 } }
I20260928 06:40:03.725265 19365 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6737b05b4a2459cbacb954cab3e5a3a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:40:03.725528 19371 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6737b05b4a2459cbacb954cab3e5a3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6737b05b4a2459cbacb954cab3e5a3a. Latest consensus state: current_term: 1 leader_uuid: "a6737b05b4a2459cbacb954cab3e5a3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6737b05b4a2459cbacb954cab3e5a3a" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41425 } } }
I20260928 06:40:03.725766 19371 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6737b05b4a2459cbacb954cab3e5a3a [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:03.725699 19372 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6737b05b4a2459cbacb954cab3e5a3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6737b05b4a2459cbacb954cab3e5a3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6737b05b4a2459cbacb954cab3e5a3a" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41425 } } }
I20260928 06:40:03.726016 19372 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6737b05b4a2459cbacb954cab3e5a3a [sys.catalog]: This master's current role is: LEADER
W20260928 06:40:03.727072 19386 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a6737b05b4a2459cbacb954cab3e5a3a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:40:03.727147 19386 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:40:03.727331 19380 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:40:03.727517 19380 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:40:03.729022 19380 catalog_manager.cc:1347] Generated new cluster ID: 0e14aae51da740ac8b85bf67fe470faf
I20260928 06:40:03.729074 19380 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:40:03 dist-test-slave-b28r krb5kdc[19284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:03.745060 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:03.722017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37756 (local address 127.18.211.126:41425)
0928 06:40:03.722224 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:03.722233 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:03.722522 (+ 289us) server_negotiation.cc:407] Connection header received
0928 06:40:03.726415 (+ 3893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:03.726422 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:03.726481 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:03.728824 (+ 2343us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:03.728976 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:03.731010 (+ 2034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:03.733162 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:03.733820 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:03.739505 (+ 5685us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:03.739597 (+ 92us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:03.739601 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:03.739601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:03.739610 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:03.742475 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:03.743099 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:03.743102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:03.743103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:03.743226 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:03.743514 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:03.743515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:03.743515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:03.743826 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:03.743947 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:03.744682 (+ 735us) server_negotiation.cc:299] Negotiation successful
0928 06:40:03.744819 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":84,"threads_started":1}
I20260928 06:40:03.763762 19380 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:40:03.764369 19380 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:40:03.771945 19380 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6737b05b4a2459cbacb954cab3e5a3a: Generated new TSK 0
I20260928 06:40:03.772123 19380 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.211.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab.
Entry for principal kudu/127.18.211.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab.
I20260928 06:40:03.797735 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.65:0
--local_ip_for_outbound_sockets=127.18.211.65
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:41425
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:41971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:03.882146 19392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:03.882352 19392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:03.882385 19392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:03.883533 19392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:03.883572 19392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:03.883608 19392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.65
I20260928 06:40:03.885005 19392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:41971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:41425
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.65
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:03.885236 19392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:03.885428 19392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:03.890022 19398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:03.890051 19400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:03.890103 19397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:03.890331 19392 server_base.cc:1034] running on GCE node
I20260928 06:40:03.890537 19392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:03.890748 19392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:03.893981 19392 hybrid_clock.cc:648] HybridClock initialized: now 1790577603893940 us; error 49 us; skew 500 ppm
Sep 28 06:40:03 dist-test-slave-b28r krb5kdc[19284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577603, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:03.897037 19392 init.cc:377] Logged in from keytab as kudu/127.18.211.65@KRBTEST.COM (short username kudu)
I20260928 06:40:03.897615 19392 webserver.cc:460] Webserver started at http://127.18.211.65:43729/ using document root <none> and password file <none>
I20260928 06:40:03.897866 19392 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:03.897927 19392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:03.898053 19392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:03.899264 19392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-0/data/instance:
uuid: "2ac5de54f83f438ebee8528ef8e6942b"
format_stamp: "Formatted at 2026-09-28 06:40:03 on dist-test-slave-b28r"
server_key: "ec7b552fb7ad4cdc93a7bf131bfdb48e"
server_key_iv: "7cdca9aa5a114c60ea37fed95879877e"
server_key_version: "encryptionkey@0"
I20260928 06:40:03.899559 19392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance:
uuid: "2ac5de54f83f438ebee8528ef8e6942b"
format_stamp: "Formatted at 2026-09-28 06:40:03 on dist-test-slave-b28r"
server_key: "ec7b552fb7ad4cdc93a7bf131bfdb48e"
server_key_iv: "7cdca9aa5a114c60ea37fed95879877e"
server_key_version: "encryptionkey@0"
I20260928 06:40:03.900754 19392 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:03.901364 19407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:03.901489 19392 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:40:03.901571 19392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-0/wal
uuid: "2ac5de54f83f438ebee8528ef8e6942b"
format_stamp: "Formatted at 2026-09-28 06:40:03 on dist-test-slave-b28r"
server_key: "ec7b552fb7ad4cdc93a7bf131bfdb48e"
server_key_iv: "7cdca9aa5a114c60ea37fed95879877e"
server_key_version: "encryptionkey@0"
I20260928 06:40:03.901639 19392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:03.929255 19392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:03.931402 19392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:03.931525 19392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:03.931761 19392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:03.932076 19392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:03.932119 19392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:03.932157 19392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:03.932181 19392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:03.937628 19392 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.65:42249
I20260928 06:40:03.937672 19520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.65:42249 every 8 connection(s)
I20260928 06:40:03.937978 19392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
I20260928 06:40:03.942965 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 19392
I20260928 06:40:03.943050 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance
I20260928 06:40:03.943220 19277 external_mini_cluster.cc:1428] Setting key c6517f059d8766f6b98d953931d79ea4
Sep 28 06:40:03 dist-test-slave-b28r krb5kdc[19284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577603, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:03.947403 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:03.938695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:49789 (local address 127.18.211.126:41425)
0928 06:40:03.938756 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:03.938759 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:03.938933 (+ 174us) server_negotiation.cc:407] Connection header received
0928 06:40:03.939497 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:03.939498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:03.939511 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:03.939565 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:03.940538 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:03.941083 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:03.942067 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:03.942268 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:03.942324 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:03.944650 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:03.944651 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:03.944652 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:03.944661 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:03.946081 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:03.946429 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:03.946430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:03.946430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:03.946501 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:03.946719 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:03.946720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:03.946720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:03.946850 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:03.946919 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:03.947164 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 06:40:03.947251 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 06:40:03.947438 19523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:03.938639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:41425 (local address 127.18.211.65:49789)
0928 06:40:03.938815 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:03.938851 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:03.939397 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:03.939642 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:03.939646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:03.940248 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:03.940442 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:03.940449 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:03.941169 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:03.941170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:03.941984 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:03.941986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:03.942337 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:03.942338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:03.942451 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:03.942929 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:03.942939 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:03.944534 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:03.946151 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:03.946157 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:03.946159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:03.946365 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:03.946557 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:03.946557 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:03.946558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:03.946653 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:03.946908 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:03.946911 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:03.947018 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 06:40:03.947142 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 06:40:03.947278 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20260928 06:40:03.948182 19521 heartbeater.cc:347] Connected to a master server at 127.18.211.126:41425
I20260928 06:40:03.948297 19521 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:03.948493 19521 heartbeater.cc:511] Master 127.18.211.126:41425 requested a full tablet report, sending...
I20260928 06:40:03.949030 19329 ts_manager.cc:194] Registered new tserver with Master: 2ac5de54f83f438ebee8528ef8e6942b (127.18.211.65:42249)
I20260928 06:40:03.949673 19329 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.65@KRBTEST.COM'} at 127.18.211.65:49789
WARNING: no policy specified for kudu/127.18.211.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab.
Entry for principal kudu/127.18.211.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab.
I20260928 06:40:03.966185 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.66:0
--local_ip_for_outbound_sockets=127.18.211.66
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:41425
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:41971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:04.046422 19526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:04.046693 19526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:04.046729 19526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:04.047922 19526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:04.047967 19526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:04.047992 19526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.66
I20260928 06:40:04.049355 19526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:41971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:41425
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.66
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:04.049533 19526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:04.049701 19526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:04.054316 19534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:04.054291 19531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:04.054514 19526 server_base.cc:1034] running on GCE node
W20260928 06:40:04.054299 19532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:04.054762 19526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:04.055009 19526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:04.056160 19526 hybrid_clock.cc:648] HybridClock initialized: now 1790577604056158 us; error 42 us; skew 500 ppm
Sep 28 06:40:04 dist-test-slave-b28r krb5kdc[19284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577604, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:04.058812 19526 init.cc:377] Logged in from keytab as kudu/127.18.211.66@KRBTEST.COM (short username kudu)
I20260928 06:40:04.059329 19526 webserver.cc:460] Webserver started at http://127.18.211.66:36873/ using document root <none> and password file <none>
I20260928 06:40:04.059526 19526 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:04.059576 19526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:04.059700 19526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:04.060729 19526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-1/data/instance:
uuid: "e725611527a544f89708edaa6e6d33cd"
format_stamp: "Formatted at 2026-09-28 06:40:04 on dist-test-slave-b28r"
server_key: "7350ea064ba70be4afe5542539cfbd94"
server_key_iv: "7c7c7bf0b1952c93f86af8f88be93260"
server_key_version: "encryptionkey@0"
I20260928 06:40:04.061048 19526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance:
uuid: "e725611527a544f89708edaa6e6d33cd"
format_stamp: "Formatted at 2026-09-28 06:40:04 on dist-test-slave-b28r"
server_key: "7350ea064ba70be4afe5542539cfbd94"
server_key_iv: "7c7c7bf0b1952c93f86af8f88be93260"
server_key_version: "encryptionkey@0"
I20260928 06:40:04.062311 19526 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:40:04.062932 19541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:04.063104 19526 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:40:04.063199 19526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-1/wal
uuid: "e725611527a544f89708edaa6e6d33cd"
format_stamp: "Formatted at 2026-09-28 06:40:04 on dist-test-slave-b28r"
server_key: "7350ea064ba70be4afe5542539cfbd94"
server_key_iv: "7c7c7bf0b1952c93f86af8f88be93260"
server_key_version: "encryptionkey@0"
I20260928 06:40:04.063258 19526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:04.076843 19526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:04.078842 19526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:04.078971 19526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:04.079217 19526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:04.079511 19526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:04.079545 19526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:04.079596 19526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:04.079620 19526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:04.085695 19526 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.66:32789
I20260928 06:40:04.085737 19654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.66:32789 every 8 connection(s)
I20260928 06:40:04.086038 19526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
I20260928 06:40:04.090633 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 19526
I20260928 06:40:04.090732 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance
I20260928 06:40:04.090909 19277 external_mini_cluster.cc:1428] Setting key 597ac02c618d21ce85cf7e0f13e597be
Sep 28 06:40:04 dist-test-slave-b28r krb5kdc[19284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577604, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:04.095588 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:04.086726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:37087 (local address 127.18.211.126:41425)
0928 06:40:04.086788 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:04.086791 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:04.086968 (+ 177us) server_negotiation.cc:407] Connection header received
0928 06:40:04.087510 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:04.087511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:04.087520 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:04.087580 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:04.088495 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:04.089013 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:04.090030 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:04.090228 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:04.090340 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:04.092638 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:04.092640 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:04.092640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:04.092655 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:04.094181 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:04.094550 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:04.094551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:04.094552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:04.094637 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:04.094912 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:04.094913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:04.094913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:04.095056 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:04.095168 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:04.095417 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 06:40:04.095497 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 06:40:04.095691 19657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:04.086702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:41425 (local address 127.18.211.66:37087)
0928 06:40:04.086855 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:04.086895 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:04.087421 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:04.087635 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:04.087638 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:04.088198 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:04.088389 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:04.088394 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:04.089102 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:04.089103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:04.089945 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:04.089948 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:04.090383 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:04.090384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:04.090518 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:04.091110 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:04.091119 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:04.092553 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:04.094264 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:04.094270 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:04.094272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:04.094480 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:04.094754 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:04.094755 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:04.094755 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:04.094858 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:04.095159 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:04.095164 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:04.095293 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 06:40:04.095380 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 06:40:04.095505 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260928 06:40:04.096145 19655 heartbeater.cc:347] Connected to a master server at 127.18.211.126:41425
I20260928 06:40:04.096238 19655 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:04.096403 19655 heartbeater.cc:511] Master 127.18.211.126:41425 requested a full tablet report, sending...
I20260928 06:40:04.096817 19329 ts_manager.cc:194] Registered new tserver with Master: e725611527a544f89708edaa6e6d33cd (127.18.211.66:32789)
I20260928 06:40:04.097306 19329 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.66@KRBTEST.COM'} at 127.18.211.66:37087
WARNING: no policy specified for kudu/127.18.211.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab.
Entry for principal kudu/127.18.211.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab.
I20260928 06:40:04.114143 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.67:0
--local_ip_for_outbound_sockets=127.18.211.67
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:41425
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:41971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:04.194199 19660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:04.194424 19660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:04.194448 19660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:04.195621 19660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:04.195662 19660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:04.195688 19660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.67
I20260928 06:40:04.197094 19660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:41971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:41425
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.67
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:04.197321 19660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:04.197539 19660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:04.202584 19666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:04.202688 19660 server_base.cc:1034] running on GCE node
W20260928 06:40:04.202596 19665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:04.202579 19668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:04.203024 19660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:04.203344 19660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:04.204511 19660 hybrid_clock.cc:648] HybridClock initialized: now 1790577604204496 us; error 23 us; skew 500 ppm
Sep 28 06:40:04 dist-test-slave-b28r krb5kdc[19284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577604, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:04.206902 19660 init.cc:377] Logged in from keytab as kudu/127.18.211.67@KRBTEST.COM (short username kudu)
I20260928 06:40:04.207404 19660 webserver.cc:460] Webserver started at http://127.18.211.67:40151/ using document root <none> and password file <none>
I20260928 06:40:04.207604 19660 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:04.207651 19660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:04.207772 19660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:04.208811 19660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-2/data/instance:
uuid: "f86272f6f5ac4c0e811d1eee12774aea"
format_stamp: "Formatted at 2026-09-28 06:40:04 on dist-test-slave-b28r"
server_key: "46c2f2341cda298ef92c8bfbce51ac92"
server_key_iv: "ad5c5552dbc1d9647b407a59de4c6c97"
server_key_version: "encryptionkey@0"
I20260928 06:40:04.209129 19660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance:
uuid: "f86272f6f5ac4c0e811d1eee12774aea"
format_stamp: "Formatted at 2026-09-28 06:40:04 on dist-test-slave-b28r"
server_key: "46c2f2341cda298ef92c8bfbce51ac92"
server_key_iv: "ad5c5552dbc1d9647b407a59de4c6c97"
server_key_version: "encryptionkey@0"
I20260928 06:40:04.210414 19660 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:40:04.211170 19675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:04.211366 19660 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:04.211457 19660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-2/wal
uuid: "f86272f6f5ac4c0e811d1eee12774aea"
format_stamp: "Formatted at 2026-09-28 06:40:04 on dist-test-slave-b28r"
server_key: "46c2f2341cda298ef92c8bfbce51ac92"
server_key_iv: "ad5c5552dbc1d9647b407a59de4c6c97"
server_key_version: "encryptionkey@0"
I20260928 06:40:04.211525 19660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:04.225387 19660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:04.227705 19660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:04.227846 19660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:04.228083 19660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:04.228387 19660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:04.228420 19660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:04.228448 19660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:04.228472 19660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:04.234227 19660 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.67:35523
I20260928 06:40:04.234293 19788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.67:35523 every 8 connection(s)
I20260928 06:40:04.234583 19660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
I20260928 06:40:04.238618 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 19660
I20260928 06:40:04.238700 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance
I20260928 06:40:04.238847 19277 external_mini_cluster.cc:1428] Setting key 6ce8d81e36f003a4d306a1d1e47b86b8
Sep 28 06:40:04 dist-test-slave-b28r krb5kdc[19284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577604, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:04.243903 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:04.235321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:53763 (local address 127.18.211.126:41425)
0928 06:40:04.235366 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:04.235368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:04.235531 (+ 163us) server_negotiation.cc:407] Connection header received
0928 06:40:04.236031 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:04.236032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:04.236043 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:04.236112 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:04.237012 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:04.237518 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:04.239054 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:04.239330 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:04.239397 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:04.241546 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:04.241548 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:04.241548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:04.241555 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:04.242646 (+ 1091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:04.243011 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:04.243012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:04.243013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:04.243069 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:04.243308 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:04.243308 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:04.243309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:04.243420 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:04.243475 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:04.243734 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 06:40:04.243813 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:40:04.244086 19791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:04.235268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:41425 (local address 127.18.211.67:53763)
0928 06:40:04.235419 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:04.235460 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:04.235928 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:04.236158 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:04.236162 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:04.236723 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:04.236918 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:04.236922 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:04.237612 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:04.237613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:04.238420 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:04.238423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:04.239405 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:04.239406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:04.239534 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:04.240054 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:04.240066 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:04.241460 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:04.242718 (+ 1258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:04.242735 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:04.242738 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:04.242946 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:04.243142 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:04.243142 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:04.243143 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:04.243252 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:04.243474 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:04.243480 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:04.243615 (+ 135us) client_negotiation.cc:769] Sending connection context
0928 06:40:04.243702 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 06:40:04.243869 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260928 06:40:04.244501 19789 heartbeater.cc:347] Connected to a master server at 127.18.211.126:41425
I20260928 06:40:04.244591 19789 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:04.244762 19789 heartbeater.cc:511] Master 127.18.211.126:41425 requested a full tablet report, sending...
I20260928 06:40:04.245163 19329 ts_manager.cc:194] Registered new tserver with Master: f86272f6f5ac4c0e811d1eee12774aea (127.18.211.67:35523)
I20260928 06:40:04.245532 19329 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.67@KRBTEST.COM'} at 127.18.211.67:53763
I20260928 06:40:04.250891 19277 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 06:40:04 dist-test-slave-b28r krb5kdc[19284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577604, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 06:40:04 dist-test-slave-b28r krb5kdc[19284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577604, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.211.65@KRBTEST.COM
I20260928 06:40:04.271912 19796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:04.264747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54946 (local address 127.18.211.65:42249)
0928 06:40:04.264883 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:04.264887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:04.264898 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:40:04.264922 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:04.264923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:04.264936 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:04.265000 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:04.265549 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:04.266304 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:04.267077 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:04.267448 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:04.267503 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:04.269559 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:04.269566 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:04.269567 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:04.269573 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:04.270943 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:04.271260 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:04.271263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:04.271263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:04.271318 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:04.271465 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:04.271465 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:04.271465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:04.271578 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:04.271693 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:04.271743 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 06:40:04.271819 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
W20260928 06:40:04.272307 19435 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54946
Sep 28 06:40:04 dist-test-slave-b28r krb5kdc[19284](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:40:04 dist-test-slave-b28r krb5kdc[19284](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577604, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.65@KRBTEST.COM
I20260928 06:40:04.293145 19796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:04.286433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54952 (local address 127.18.211.65:42249)
0928 06:40:04.286499 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:04.286501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:04.286555 (+ 54us) server_negotiation.cc:407] Connection header received
0928 06:40:04.286607 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:04.286608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:04.286620 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:04.286672 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:04.287254 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:04.287747 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:04.288537 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:04.288747 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:04.288803 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:04.290921 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:04.290922 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:04.290923 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:04.290929 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:04.292202 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:04.292503 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:04.292504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:04.292504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:04.292555 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:04.292720 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:04.292721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:04.292722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:04.292835 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:04.292940 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:04.293016 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 06:40:04.293079 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:40:04.293704 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 19392
I20260928 06:40:04.297830 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 19526
I20260928 06:40:04.301766 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 19660
I20260928 06:40:04.305393 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 19297
2026-09-28T06:40:04Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1788 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:40:04 dist-test-slave-b28r krb5kdc[19803](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:40:04 dist-test-slave-b28r krb5kdc[19803](info): set up 2 sockets
Sep 28 06:40:04 dist-test-slave-b28r krb5kdc[19803](info): commencing operation
krb5kdc: starting...
W20260928 06:40:05.251605 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:40:05 dist-test-slave-b28r krb5kdc[19803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:40:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:40:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.211.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
I20260928 06:40:05.360070 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:37789
--webserver_interface=127.18.211.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:46659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:37789
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:05.438328 19816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:05.438539 19816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:05.438570 19816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:05.439842 19816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:40:05.439898 19816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:05.439916 19816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:05.439932 19816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:40:05.439946 19816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:40:05.441442 19816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:46659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:37789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:37789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:05.441830 19816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:05.442056 19816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:40:05.446916 19816 server_base.cc:1034] running on GCE node
W20260928 06:40:05.446856 19821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:05.446882 19824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:05.446856 19822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:05.447361 19816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:05.447642 19816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:05.448810 19816 hybrid_clock.cc:648] HybridClock initialized: now 1790577605448792 us; error 34 us; skew 500 ppm
Sep 28 06:40:05 dist-test-slave-b28r krb5kdc[19803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577605, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:05.451431 19816 init.cc:377] Logged in from keytab as kudu/127.18.211.126@KRBTEST.COM (short username kudu)
I20260928 06:40:05.451898 19816 webserver.cc:460] Webserver started at http://127.18.211.126:40319/ using document root <none> and password file <none>
I20260928 06:40:05.452107 19816 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:05.452170 19816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:05.452313 19816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:05.453413 19816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "fa8365b746064f3294471442ea410d8e"
format_stamp: "Formatted at 2026-09-28 06:40:05 on dist-test-slave-b28r"
server_key: "ac0c532af9cf5fd6d134ad3209e55041"
server_key_iv: "6f2b1c8993c9d8f43d57ccf894cfb994"
server_key_version: "encryptionkey@0"
I20260928 06:40:05.453787 19816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "fa8365b746064f3294471442ea410d8e"
format_stamp: "Formatted at 2026-09-28 06:40:05 on dist-test-slave-b28r"
server_key: "ac0c532af9cf5fd6d134ad3209e55041"
server_key_iv: "6f2b1c8993c9d8f43d57ccf894cfb994"
server_key_version: "encryptionkey@0"
I20260928 06:40:05.455224 19816 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:40:05.455996 19831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:05.456192 19816 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:05.456283 19816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "fa8365b746064f3294471442ea410d8e"
format_stamp: "Formatted at 2026-09-28 06:40:05 on dist-test-slave-b28r"
server_key: "ac0c532af9cf5fd6d134ad3209e55041"
server_key_iv: "6f2b1c8993c9d8f43d57ccf894cfb994"
server_key_version: "encryptionkey@0"
I20260928 06:40:05.456357 19816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:05.462793 19816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:05.465011 19816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:05.465162 19816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:05.469331 19816 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:37789
I20260928 06:40:05.469359 19883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:37789 every 8 connection(s)
I20260928 06:40:05.469707 19816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:40:05.470563 19884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:05.472832 19884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa8365b746064f3294471442ea410d8e: Bootstrap starting.
I20260928 06:40:05.473351 19884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa8365b746064f3294471442ea410d8e: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:05.473573 19884 log.cc:824] T 00000000000000000000000000000000 P fa8365b746064f3294471442ea410d8e: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:05.474306 19884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa8365b746064f3294471442ea410d8e: No bootstrap required, opened a new log
I20260928 06:40:05.474373 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 19816
I20260928 06:40:05.474462 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:40:05.474609 19277 external_mini_cluster.cc:1428] Setting key 86267900d3e575fcfb1e871823cf7a6b
I20260928 06:40:05.476154 19884 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa8365b746064f3294471442ea410d8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa8365b746064f3294471442ea410d8e" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 37789 } }
I20260928 06:40:05.476298 19884 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa8365b746064f3294471442ea410d8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:05.476331 19884 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa8365b746064f3294471442ea410d8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa8365b746064f3294471442ea410d8e, State: Initialized, Role: FOLLOWER
I20260928 06:40:05.476455 19884 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa8365b746064f3294471442ea410d8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa8365b746064f3294471442ea410d8e" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 37789 } }
I20260928 06:40:05.476524 19884 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa8365b746064f3294471442ea410d8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:40:05.476555 19884 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa8365b746064f3294471442ea410d8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:40:05.476601 19884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa8365b746064f3294471442ea410d8e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:05.477430 19884 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa8365b746064f3294471442ea410d8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa8365b746064f3294471442ea410d8e" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 37789 } }
I20260928 06:40:05.477527 19884 leader_election.cc:302] T 00000000000000000000000000000000 P fa8365b746064f3294471442ea410d8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa8365b746064f3294471442ea410d8e; no voters:
I20260928 06:40:05.477679 19884 leader_election.cc:288] T 00000000000000000000000000000000 P fa8365b746064f3294471442ea410d8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:40:05.477718 19889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa8365b746064f3294471442ea410d8e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:05.477830 19889 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa8365b746064f3294471442ea410d8e [term 1 LEADER]: Becoming Leader. State: Replica: fa8365b746064f3294471442ea410d8e, State: Running, Role: LEADER
I20260928 06:40:05.477926 19884 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa8365b746064f3294471442ea410d8e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:40:05.477908 19889 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa8365b746064f3294471442ea410d8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa8365b746064f3294471442ea410d8e" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 37789 } }
I20260928 06:40:05.478240 19889 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa8365b746064f3294471442ea410d8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa8365b746064f3294471442ea410d8e. Latest consensus state: current_term: 1 leader_uuid: "fa8365b746064f3294471442ea410d8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa8365b746064f3294471442ea410d8e" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 37789 } } }
I20260928 06:40:05.478305 19889 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa8365b746064f3294471442ea410d8e [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:05.478513 19894 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:40:05.479018 19894 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:40:05.481572 19891 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa8365b746064f3294471442ea410d8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa8365b746064f3294471442ea410d8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa8365b746064f3294471442ea410d8e" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 37789 } } }
I20260928 06:40:05.481685 19891 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa8365b746064f3294471442ea410d8e [sys.catalog]: This master's current role is: LEADER
Sep 28 06:40:05 dist-test-slave-b28r krb5kdc[19803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:05.482113 19894 catalog_manager.cc:1347] Generated new cluster ID: 83e8d7b4b76348b28fb9efea38157ac5
I20260928 06:40:05.482173 19894 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:40:05.485961 19888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:05.475144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39654 (local address 127.18.211.126:37789)
0928 06:40:05.475317 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:05.475325 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:05.475350 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:40:05.476133 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:05.476140 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:05.476231 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:05.476362 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:05.477014 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:05.478064 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:05.479142 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:05.479674 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:05.479774 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:05.482467 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:05.482471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:05.482471 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:05.482483 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:05.484396 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:05.484766 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:05.484769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:05.484770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:05.484903 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:05.485076 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:05.485077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:05.485077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:05.485383 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:05.485533 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:05.485610 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 06:40:05.485762 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
I20260928 06:40:05.521564 19894 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:40:05.522317 19894 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:40:05.526808 19894 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa8365b746064f3294471442ea410d8e: Generated new TSK 0
I20260928 06:40:05.526988 19894 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.211.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab.
Entry for principal kudu/127.18.211.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab.
I20260928 06:40:05.553053 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.65:0
--local_ip_for_outbound_sockets=127.18.211.65
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:37789
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:46659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:05.632138 19911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:05.632355 19911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:05.632380 19911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:05.633504 19911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:05.633543 19911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:05.633569 19911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.65
I20260928 06:40:05.635164 19911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:46659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:37789
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.65
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:05.635352 19911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:05.635527 19911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:05.640327 19917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:05.640278 19919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:05.640416 19911 server_base.cc:1034] running on GCE node
W20260928 06:40:05.640295 19916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:05.640719 19911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:05.640962 19911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:05.642112 19911 hybrid_clock.cc:648] HybridClock initialized: now 1790577605642101 us; error 31 us; skew 500 ppm
Sep 28 06:40:05 dist-test-slave-b28r krb5kdc[19803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577605, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:05.644486 19911 init.cc:377] Logged in from keytab as kudu/127.18.211.65@KRBTEST.COM (short username kudu)
I20260928 06:40:05.644896 19911 webserver.cc:460] Webserver started at http://127.18.211.65:40623/ using document root <none> and password file <none>
I20260928 06:40:05.645087 19911 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:05.645128 19911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:05.645253 19911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:05.646248 19911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-0/data/instance:
uuid: "c5e5e90843a445c8be90cd1aecf655ca"
format_stamp: "Formatted at 2026-09-28 06:40:05 on dist-test-slave-b28r"
server_key: "493582ef274612b80517e60e017d0be8"
server_key_iv: "a1b17fe3f829fcd5fd8a6784bd0083dd"
server_key_version: "encryptionkey@0"
I20260928 06:40:05.646560 19911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance:
uuid: "c5e5e90843a445c8be90cd1aecf655ca"
format_stamp: "Formatted at 2026-09-28 06:40:05 on dist-test-slave-b28r"
server_key: "493582ef274612b80517e60e017d0be8"
server_key_iv: "a1b17fe3f829fcd5fd8a6784bd0083dd"
server_key_version: "encryptionkey@0"
I20260928 06:40:05.647912 19911 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:40:05.648718 19926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:05.648871 19911 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:40:05.648957 19911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-0/wal
uuid: "c5e5e90843a445c8be90cd1aecf655ca"
format_stamp: "Formatted at 2026-09-28 06:40:05 on dist-test-slave-b28r"
server_key: "493582ef274612b80517e60e017d0be8"
server_key_iv: "a1b17fe3f829fcd5fd8a6784bd0083dd"
server_key_version: "encryptionkey@0"
I20260928 06:40:05.649019 19911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:05.663451 19911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:05.665467 19911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:05.665608 19911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:05.665824 19911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:05.666116 19911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:05.666155 19911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:05.666194 19911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:05.666213 19911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:05.671564 19911 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.65:42919
I20260928 06:40:05.671626 20039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.65:42919 every 8 connection(s)
I20260928 06:40:05.671911 19911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
I20260928 06:40:05.677389 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 19911
I20260928 06:40:05.677482 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance
I20260928 06:40:05.677629 19277 external_mini_cluster.cc:1428] Setting key 631fa8c50d6c38922f3dcc242b5721c2
Sep 28 06:40:05 dist-test-slave-b28r krb5kdc[19803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577605, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:05.681542 19888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:05.672610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:51413 (local address 127.18.211.126:37789)
0928 06:40:05.672685 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:05.672687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:05.672854 (+ 167us) server_negotiation.cc:407] Connection header received
0928 06:40:05.673803 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:05.673804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:05.673821 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:05.673872 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:05.674478 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:05.674977 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:05.676001 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:05.676207 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:05.676261 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:05.678646 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:05.678648 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:05.678648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:05.678656 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:05.680181 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:05.680582 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:05.680584 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:05.680584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:05.680646 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:05.680882 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:05.680883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:05.680884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:05.681004 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:05.681068 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:05.681335 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 06:40:05.681430 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 06:40:05.681701 20042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:05.672536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:37789 (local address 127.18.211.65:51413)
0928 06:40:05.672722 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:05.672767 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:05.673700 (+ 933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:05.673943 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:05.673947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:05.674208 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:05.674391 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:05.674395 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:05.675056 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:05.675057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:05.675913 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:05.675915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:05.676274 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:05.676275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:05.676394 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:05.676847 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:05.676855 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:05.678538 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:05.680266 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:05.680274 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:05.680276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:05.680507 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:05.680706 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:05.680707 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:05.680707 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:05.680822 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:05.681063 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:05.681070 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:05.681210 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 06:40:05.681305 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 06:40:05.681464 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":80,"threads_started":1}
I20260928 06:40:05.682202 20040 heartbeater.cc:347] Connected to a master server at 127.18.211.126:37789
I20260928 06:40:05.682310 20040 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:05.682504 20040 heartbeater.cc:511] Master 127.18.211.126:37789 requested a full tablet report, sending...
I20260928 06:40:05.683071 19848 ts_manager.cc:194] Registered new tserver with Master: c5e5e90843a445c8be90cd1aecf655ca (127.18.211.65:42919)
I20260928 06:40:05.683979 19848 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.65@KRBTEST.COM'} at 127.18.211.65:51413
WARNING: no policy specified for kudu/127.18.211.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab.
Entry for principal kudu/127.18.211.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab.
I20260928 06:40:05.699575 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.66:0
--local_ip_for_outbound_sockets=127.18.211.66
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:37789
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:46659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:05.777987 20045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:05.778162 20045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:05.778189 20045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:05.779340 20045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:05.779379 20045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:05.779403 20045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.66
I20260928 06:40:05.780826 20045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:46659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:37789
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.66
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:05.781026 20045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:05.781231 20045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:05.785732 20051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:05.785861 20050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:05.785903 20045 server_base.cc:1034] running on GCE node
W20260928 06:40:05.785755 20053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:05.786212 20045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:05.786398 20045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:05.787528 20045 hybrid_clock.cc:648] HybridClock initialized: now 1790577605787511 us; error 34 us; skew 500 ppm
Sep 28 06:40:05 dist-test-slave-b28r krb5kdc[19803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577605, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:05.789799 20045 init.cc:377] Logged in from keytab as kudu/127.18.211.66@KRBTEST.COM (short username kudu)
I20260928 06:40:05.790241 20045 webserver.cc:460] Webserver started at http://127.18.211.66:38677/ using document root <none> and password file <none>
I20260928 06:40:05.790438 20045 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:05.790483 20045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:05.790603 20045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:05.791577 20045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-1/data/instance:
uuid: "6767e2d81a5146dd94d65223fa008047"
format_stamp: "Formatted at 2026-09-28 06:40:05 on dist-test-slave-b28r"
server_key: "aebadcaa9b708459db7d8a4ff3876ead"
server_key_iv: "ffd175efa33b71c7cc1b95cc77c1db33"
server_key_version: "encryptionkey@0"
I20260928 06:40:05.791877 20045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance:
uuid: "6767e2d81a5146dd94d65223fa008047"
format_stamp: "Formatted at 2026-09-28 06:40:05 on dist-test-slave-b28r"
server_key: "aebadcaa9b708459db7d8a4ff3876ead"
server_key_iv: "ffd175efa33b71c7cc1b95cc77c1db33"
server_key_version: "encryptionkey@0"
I20260928 06:40:05.793028 20045 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:40:05.793701 20060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:05.793848 20045 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:40:05.793924 20045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-1/wal
uuid: "6767e2d81a5146dd94d65223fa008047"
format_stamp: "Formatted at 2026-09-28 06:40:05 on dist-test-slave-b28r"
server_key: "aebadcaa9b708459db7d8a4ff3876ead"
server_key_iv: "ffd175efa33b71c7cc1b95cc77c1db33"
server_key_version: "encryptionkey@0"
I20260928 06:40:05.793979 20045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:05.808782 20045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:05.810456 20045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:05.810578 20045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:05.810771 20045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:05.811038 20045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:05.811069 20045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:05.811120 20045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:05.811144 20045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:05.816494 20045 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.66:45723
I20260928 06:40:05.816550 20173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.66:45723 every 8 connection(s)
I20260928 06:40:05.816813 20045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:40:05 dist-test-slave-b28r krb5kdc[19803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577605, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:05.823590 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 20045
I20260928 06:40:05.823663 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance
I20260928 06:40:05.823834 19277 external_mini_cluster.cc:1428] Setting key 8490f680b15aae73f157a065d9ad4487
I20260928 06:40:05.826525 19888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:05.817464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:47883 (local address 127.18.211.126:37789)
0928 06:40:05.817523 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:05.817525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:05.817710 (+ 185us) server_negotiation.cc:407] Connection header received
0928 06:40:05.818575 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:05.818576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:05.818588 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:05.818659 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:05.819252 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:05.819760 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:05.820719 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:05.820913 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:05.820971 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:05.822885 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:05.822886 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:05.822887 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:05.822893 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:05.824524 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:05.825616 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:05.825617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:05.825617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:05.825673 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:05.825917 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:05.825917 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:05.825917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:05.826024 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:05.826082 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:05.826354 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 06:40:05.826438 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:40:05.826730 20176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:05.817428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:37789 (local address 127.18.211.66:47883)
0928 06:40:05.817577 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:05.817617 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:05.818481 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:05.818720 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:05.818723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:05.818955 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:05.819169 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:05.819175 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:05.819848 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:05.819849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:05.820636 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:05.820638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:05.820982 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:05.820983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:05.821079 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:05.821492 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:05.821500 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:05.822798 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:05.825156 (+ 2358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:05.825164 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:05.825166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:05.825525 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:05.825737 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:05.825738 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:05.825739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:05.825863 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:05.826087 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:05.826094 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:05.826231 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 06:40:05.826323 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 06:40:05.826473 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
I20260928 06:40:05.827143 20174 heartbeater.cc:347] Connected to a master server at 127.18.211.126:37789
I20260928 06:40:05.827239 20174 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:05.827410 20174 heartbeater.cc:511] Master 127.18.211.126:37789 requested a full tablet report, sending...
I20260928 06:40:05.827792 19848 ts_manager.cc:194] Registered new tserver with Master: 6767e2d81a5146dd94d65223fa008047 (127.18.211.66:45723)
I20260928 06:40:05.828274 19848 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.66@KRBTEST.COM'} at 127.18.211.66:47883
WARNING: no policy specified for kudu/127.18.211.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab.
Entry for principal kudu/127.18.211.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab.
I20260928 06:40:05.846952 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.67:0
--local_ip_for_outbound_sockets=127.18.211.67
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:37789
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:46659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:05.927395 20179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:05.927590 20179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:05.927613 20179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:05.928714 20179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:05.928757 20179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:05.928782 20179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.67
I20260928 06:40:05.930202 20179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:46659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:37789
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.67
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:05.930392 20179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:05.930564 20179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:05.935099 20187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:05.935117 20185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:05.935120 20184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:05.935381 20179 server_base.cc:1034] running on GCE node
I20260928 06:40:05.935578 20179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:05.935783 20179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:05.936910 20179 hybrid_clock.cc:648] HybridClock initialized: now 1790577605936897 us; error 27 us; skew 500 ppm
Sep 28 06:40:05 dist-test-slave-b28r krb5kdc[19803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577605, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:05.939374 20179 init.cc:377] Logged in from keytab as kudu/127.18.211.67@KRBTEST.COM (short username kudu)
I20260928 06:40:05.939742 20179 webserver.cc:460] Webserver started at http://127.18.211.67:35795/ using document root <none> and password file <none>
I20260928 06:40:05.939944 20179 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:05.939991 20179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:05.940116 20179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:05.941216 20179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-2/data/instance:
uuid: "ce17261982124a17b372dc06bba842a7"
format_stamp: "Formatted at 2026-09-28 06:40:05 on dist-test-slave-b28r"
server_key: "0fe45a8560c0338de623be35117cd8a6"
server_key_iv: "de35db7c95fb991dfde9783d697df71c"
server_key_version: "encryptionkey@0"
I20260928 06:40:05.941527 20179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance:
uuid: "ce17261982124a17b372dc06bba842a7"
format_stamp: "Formatted at 2026-09-28 06:40:05 on dist-test-slave-b28r"
server_key: "0fe45a8560c0338de623be35117cd8a6"
server_key_iv: "de35db7c95fb991dfde9783d697df71c"
server_key_version: "encryptionkey@0"
I20260928 06:40:05.942698 20179 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:40:05.943391 20194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:05.943521 20179 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:05.943599 20179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-2/wal
uuid: "ce17261982124a17b372dc06bba842a7"
format_stamp: "Formatted at 2026-09-28 06:40:05 on dist-test-slave-b28r"
server_key: "0fe45a8560c0338de623be35117cd8a6"
server_key_iv: "de35db7c95fb991dfde9783d697df71c"
server_key_version: "encryptionkey@0"
I20260928 06:40:05.943655 20179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:05.966421 20179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:05.968631 20179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:05.968772 20179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:05.968981 20179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:05.969295 20179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:05.969329 20179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:05.969357 20179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:05.969381 20179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:05.974784 20179 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.67:40111
I20260928 06:40:05.974859 20307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.67:40111 every 8 connection(s)
I20260928 06:40:05.975142 20179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:40:05 dist-test-slave-b28r krb5kdc[19803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577605, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:05.981397 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 20179
I20260928 06:40:05.981505 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance
I20260928 06:40:05.981710 19277 external_mini_cluster.cc:1428] Setting key 25ce70af4aea19a7cc09941f3b56f28c
I20260928 06:40:05.984387 19888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:05.975809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:35349 (local address 127.18.211.126:37789)
0928 06:40:05.975880 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:05.975882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:05.976032 (+ 150us) server_negotiation.cc:407] Connection header received
0928 06:40:05.976908 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:05.976909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:05.976918 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:05.976962 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:05.977522 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:05.978078 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:05.979107 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:05.979365 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:05.979421 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:05.981573 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:05.981574 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:05.981575 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:05.981583 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:05.983024 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:05.983450 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:05.983451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:05.983452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:05.983516 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:05.983749 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:05.983750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:05.983750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:05.983873 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:05.983937 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:05.984211 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 06:40:05.984295 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:40:05.984555 20310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:05.975770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:37789 (local address 127.18.211.67:35349)
0928 06:40:05.975919 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:05.975955 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:05.976829 (+ 874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:05.977026 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:05.977029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:05.977269 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:05.977452 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:05.977457 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:05.978150 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:05.978151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:05.979012 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:05.979015 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:05.979442 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:05.979443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:05.979543 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:05.979930 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:05.979938 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:05.981474 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:05.983145 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:05.983152 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:05.983154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:05.983377 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:05.983576 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:05.983577 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:05.983577 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:05.983688 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:05.983933 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:05.983939 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:05.984077 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 06:40:05.984181 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 06:40:05.984339 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":78,"threads_started":1}
I20260928 06:40:05.984984 20308 heartbeater.cc:347] Connected to a master server at 127.18.211.126:37789
I20260928 06:40:05.985069 20308 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:05.985246 20308 heartbeater.cc:511] Master 127.18.211.126:37789 requested a full tablet report, sending...
I20260928 06:40:05.985580 19848 ts_manager.cc:194] Registered new tserver with Master: ce17261982124a17b372dc06bba842a7 (127.18.211.67:40111)
I20260928 06:40:05.985963 19848 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.67@KRBTEST.COM'} at 127.18.211.67:35349
I20260928 06:40:05.993280 19277 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:40:06.000464 19888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:05.994872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39660 (local address 127.18.211.126:37789)
0928 06:40:05.994918 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:05.994920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:05.995033 (+ 113us) server_negotiation.cc:407] Connection header received
0928 06:40:05.995072 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:05.995072 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:05.995100 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:05.995155 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:05.995674 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:05.996147 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:05.996908 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:05.997089 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:05.997144 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:05.998283 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:05.998284 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:05.998284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:05.998290 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:05.999526 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:05.999852 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:05.999853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:05.999853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:05.999900 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:06.000037 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:06.000038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:06.000038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:06.000160 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:06.000226 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:06.000342 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 06:40:06.000399 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:40:06.002620 19848 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39660:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:40:06.004251 19848 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:40:06.010432 20319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:06.006299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55592 (local address 127.18.211.67:40111)
0928 06:40:06.006460 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:06.006466 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:06.006773 (+ 307us) server_negotiation.cc:407] Connection header received
0928 06:40:06.006849 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:06.006851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:06.006867 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:06.006977 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:06.007143 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:06.008266 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:06.009332 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:06.010015 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:06.010092 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:06.010150 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:06.010306 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 06:40:06.010351 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":86,"threads_started":1}
I20260928 06:40:06.010465 20318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:06.006208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:40111 (local address 127.0.0.1:55592)
0928 06:40:06.006642 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:06.006680 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:06.006764 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:06.006985 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:06.006988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:06.006989 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:06.007059 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:06.007061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:06.008409 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:06.008410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:06.009204 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:06.009207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:06.010117 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:06.010118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:06.010194 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:06.010205 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 06:40:06.010289 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 06:40:06.010388 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":74,"threads_started":1}
I20260928 06:40:06.010644 20321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:06.006474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57350 (local address 127.18.211.66:45723)
0928 06:40:06.006624 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:06.006627 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:06.006931 (+ 304us) server_negotiation.cc:407] Connection header received
0928 06:40:06.006979 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:06.006982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:06.006997 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:06.007066 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:06.007337 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:06.008041 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:06.009521 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:06.010223 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:06.010286 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:06.010341 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:06.010520 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 06:40:06.010565 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":82,"threads_started":1}
I20260928 06:40:06.010824 20317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:06.006267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:45723 (local address 127.0.0.1:57350)
0928 06:40:06.006866 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:06.006874 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:06.006916 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:06.007226 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:06.007227 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:06.007230 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:06.007264 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:06.007266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:06.008137 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:06.008138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:06.009435 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:06.009438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:06.010324 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:06.010325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:06.010382 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:06.010386 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:40:06.010439 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 06:40:06.010474 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":4,"thread_start_us":52,"threads_started":1}
I20260928 06:40:06.011443 20316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:06.006213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:42919 (local address 127.0.0.1:55900)
0928 06:40:06.006722 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:06.006731 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:06.006777 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:06.007050 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:06.007051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:06.007053 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:06.007150 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:06.007152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:06.008380 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:06.008381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:06.009453 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:06.009455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:06.011215 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:06.011216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:06.011279 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:06.011283 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:40:06.011339 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 06:40:06.011380 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":60,"threads_started":1}
I20260928 06:40:06.011512 20242 tablet_service.cc:1467] Processing CreateTablet for tablet 0a37e1cc7e6945cf90d08d32962eaeb1 (DEFAULT_TABLE table=test-table [id=6af655d4b08944f2945c661ae5e5752f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:06.011502 20108 tablet_service.cc:1467] Processing CreateTablet for tablet 0a37e1cc7e6945cf90d08d32962eaeb1 (DEFAULT_TABLE table=test-table [id=6af655d4b08944f2945c661ae5e5752f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:06.011735 20320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:06.006473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55900 (local address 127.18.211.65:42919)
0928 06:40:06.006698 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:06.006700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:06.006772 (+ 72us) server_negotiation.cc:407] Connection header received
0928 06:40:06.006849 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:06.006851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:06.006867 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:06.006951 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:06.007242 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:06.008293 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:06.010377 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:06.011134 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:06.011203 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:06.011252 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:06.011570 (+ 318us) server_negotiation.cc:299] Negotiation successful
0928 06:40:06.011616 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":79,"threads_started":1}
I20260928 06:40:06.011844 20242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a37e1cc7e6945cf90d08d32962eaeb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:06.011844 20108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a37e1cc7e6945cf90d08d32962eaeb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:06.014250 20323 tablet_bootstrap.cc:492] T 0a37e1cc7e6945cf90d08d32962eaeb1 P 6767e2d81a5146dd94d65223fa008047: Bootstrap starting.
I20260928 06:40:06.014546 20322 tablet_bootstrap.cc:492] T 0a37e1cc7e6945cf90d08d32962eaeb1 P ce17261982124a17b372dc06bba842a7: Bootstrap starting.
I20260928 06:40:06.014950 20323 tablet_bootstrap.cc:654] T 0a37e1cc7e6945cf90d08d32962eaeb1 P 6767e2d81a5146dd94d65223fa008047: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:06.015174 20322 tablet_bootstrap.cc:654] T 0a37e1cc7e6945cf90d08d32962eaeb1 P ce17261982124a17b372dc06bba842a7: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:06.015146 19974 tablet_service.cc:1467] Processing CreateTablet for tablet 0a37e1cc7e6945cf90d08d32962eaeb1 (DEFAULT_TABLE table=test-table [id=6af655d4b08944f2945c661ae5e5752f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:06.015511 19974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a37e1cc7e6945cf90d08d32962eaeb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:06.015563 20323 log.cc:824] T 0a37e1cc7e6945cf90d08d32962eaeb1 P 6767e2d81a5146dd94d65223fa008047: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:06.015703 20322 log.cc:824] T 0a37e1cc7e6945cf90d08d32962eaeb1 P ce17261982124a17b372dc06bba842a7: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:06.016390 20322 tablet_bootstrap.cc:492] T 0a37e1cc7e6945cf90d08d32962eaeb1 P ce17261982124a17b372dc06bba842a7: No bootstrap required, opened a new log
I20260928 06:40:06.016456 20322 ts_tablet_manager.cc:1397] T 0a37e1cc7e6945cf90d08d32962eaeb1 P ce17261982124a17b372dc06bba842a7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 06:40:06.018007 20322 raft_consensus.cc:348] T 0a37e1cc7e6945cf90d08d32962eaeb1 P ce17261982124a17b372dc06bba842a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce17261982124a17b372dc06bba842a7" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40111 } } peers { permanent_uuid: "c5e5e90843a445c8be90cd1aecf655ca" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 42919 } } peers { permanent_uuid: "6767e2d81a5146dd94d65223fa008047" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 45723 } }
I20260928 06:40:06.018206 20322 raft_consensus.cc:374] T 0a37e1cc7e6945cf90d08d32962eaeb1 P ce17261982124a17b372dc06bba842a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:06.018257 20322 raft_consensus.cc:729] T 0a37e1cc7e6945cf90d08d32962eaeb1 P ce17261982124a17b372dc06bba842a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce17261982124a17b372dc06bba842a7, State: Initialized, Role: FOLLOWER
I20260928 06:40:06.018360 20326 tablet_bootstrap.cc:492] T 0a37e1cc7e6945cf90d08d32962eaeb1 P c5e5e90843a445c8be90cd1aecf655ca: Bootstrap starting.
I20260928 06:40:06.018430 20322 consensus_queue.cc:260] T 0a37e1cc7e6945cf90d08d32962eaeb1 P ce17261982124a17b372dc06bba842a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce17261982124a17b372dc06bba842a7" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40111 } } peers { permanent_uuid: "c5e5e90843a445c8be90cd1aecf655ca" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 42919 } } peers { permanent_uuid: "6767e2d81a5146dd94d65223fa008047" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 45723 } }
I20260928 06:40:06.018736 20308 heartbeater.cc:503] Master 127.18.211.126:37789 was elected leader, sending a full tablet report...
I20260928 06:40:06.018745 20322 ts_tablet_manager.cc:1428] T 0a37e1cc7e6945cf90d08d32962eaeb1 P ce17261982124a17b372dc06bba842a7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:40:06.019039 20326 tablet_bootstrap.cc:654] T 0a37e1cc7e6945cf90d08d32962eaeb1 P c5e5e90843a445c8be90cd1aecf655ca: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:06.019120 20323 tablet_bootstrap.cc:492] T 0a37e1cc7e6945cf90d08d32962eaeb1 P 6767e2d81a5146dd94d65223fa008047: No bootstrap required, opened a new log
I20260928 06:40:06.019194 20323 ts_tablet_manager.cc:1397] T 0a37e1cc7e6945cf90d08d32962eaeb1 P 6767e2d81a5146dd94d65223fa008047: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260928 06:40:06.019562 20326 log.cc:824] T 0a37e1cc7e6945cf90d08d32962eaeb1 P c5e5e90843a445c8be90cd1aecf655ca: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:06.020340 20326 tablet_bootstrap.cc:492] T 0a37e1cc7e6945cf90d08d32962eaeb1 P c5e5e90843a445c8be90cd1aecf655ca: No bootstrap required, opened a new log
I20260928 06:40:06.020417 20326 ts_tablet_manager.cc:1397] T 0a37e1cc7e6945cf90d08d32962eaeb1 P c5e5e90843a445c8be90cd1aecf655ca: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:40:06.021243 20323 raft_consensus.cc:348] T 0a37e1cc7e6945cf90d08d32962eaeb1 P 6767e2d81a5146dd94d65223fa008047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce17261982124a17b372dc06bba842a7" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40111 } } peers { permanent_uuid: "c5e5e90843a445c8be90cd1aecf655ca" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 42919 } } peers { permanent_uuid: "6767e2d81a5146dd94d65223fa008047" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 45723 } }
I20260928 06:40:06.021454 20323 raft_consensus.cc:374] T 0a37e1cc7e6945cf90d08d32962eaeb1 P 6767e2d81a5146dd94d65223fa008047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:06.021494 20323 raft_consensus.cc:729] T 0a37e1cc7e6945cf90d08d32962eaeb1 P 6767e2d81a5146dd94d65223fa008047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6767e2d81a5146dd94d65223fa008047, State: Initialized, Role: FOLLOWER
I20260928 06:40:06.021616 20323 consensus_queue.cc:260] T 0a37e1cc7e6945cf90d08d32962eaeb1 P 6767e2d81a5146dd94d65223fa008047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce17261982124a17b372dc06bba842a7" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40111 } } peers { permanent_uuid: "c5e5e90843a445c8be90cd1aecf655ca" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 42919 } } peers { permanent_uuid: "6767e2d81a5146dd94d65223fa008047" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 45723 } }
I20260928 06:40:06.021919 20174 heartbeater.cc:503] Master 127.18.211.126:37789 was elected leader, sending a full tablet report...
I20260928 06:40:06.021915 20323 ts_tablet_manager.cc:1428] T 0a37e1cc7e6945cf90d08d32962eaeb1 P 6767e2d81a5146dd94d65223fa008047: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 06:40:06.022439 20326 raft_consensus.cc:348] T 0a37e1cc7e6945cf90d08d32962eaeb1 P c5e5e90843a445c8be90cd1aecf655ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce17261982124a17b372dc06bba842a7" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40111 } } peers { permanent_uuid: "c5e5e90843a445c8be90cd1aecf655ca" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 42919 } } peers { permanent_uuid: "6767e2d81a5146dd94d65223fa008047" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 45723 } }
I20260928 06:40:06.022607 20326 raft_consensus.cc:374] T 0a37e1cc7e6945cf90d08d32962eaeb1 P c5e5e90843a445c8be90cd1aecf655ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:06.022642 20326 raft_consensus.cc:729] T 0a37e1cc7e6945cf90d08d32962eaeb1 P c5e5e90843a445c8be90cd1aecf655ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5e5e90843a445c8be90cd1aecf655ca, State: Initialized, Role: FOLLOWER
I20260928 06:40:06.022753 20326 consensus_queue.cc:260] T 0a37e1cc7e6945cf90d08d32962eaeb1 P c5e5e90843a445c8be90cd1aecf655ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce17261982124a17b372dc06bba842a7" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40111 } } peers { permanent_uuid: "c5e5e90843a445c8be90cd1aecf655ca" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 42919 } } peers { permanent_uuid: "6767e2d81a5146dd94d65223fa008047" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 45723 } }
I20260928 06:40:06.023029 20326 ts_tablet_manager.cc:1428] T 0a37e1cc7e6945cf90d08d32962eaeb1 P c5e5e90843a445c8be90cd1aecf655ca: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:40:06.023047 20040 heartbeater.cc:503] Master 127.18.211.126:37789 was elected leader, sending a full tablet report...
I20260928 06:40:06.053787 20327 raft_consensus.cc:482] T 0a37e1cc7e6945cf90d08d32962eaeb1 P ce17261982124a17b372dc06bba842a7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:40:06.053937 20327 raft_consensus.cc:504] T 0a37e1cc7e6945cf90d08d32962eaeb1 P ce17261982124a17b372dc06bba842a7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce17261982124a17b372dc06bba842a7" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40111 } } peers { permanent_uuid: "c5e5e90843a445c8be90cd1aecf655ca" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 42919 } } peers { permanent_uuid: "6767e2d81a5146dd94d65223fa008047" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 45723 } }
I20260928 06:40:06.054272 20327 leader_election.cc:288] T 0a37e1cc7e6945cf90d08d32962eaeb1 P ce17261982124a17b372dc06bba842a7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5e5e90843a445c8be90cd1aecf655ca (127.18.211.65:42919), 6767e2d81a5146dd94d65223fa008047 (127.18.211.66:45723)
I20260928 06:40:06.057060 20321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:06.054586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:34995 (local address 127.18.211.66:45723)
0928 06:40:06.054644 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:06.054647 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:06.054655 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:40:06.054682 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:06.054683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:06.054693 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:06.054770 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:06.054915 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:06.055438 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:06.056379 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:06.056754 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:06.056813 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:06.056821 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:06.056951 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 06:40:06.056986 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 06:40:06.057060 20320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:06.054532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:45713 (local address 127.18.211.65:42919)
0928 06:40:06.054585 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:06.054587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:06.054667 (+ 80us) server_negotiation.cc:407] Connection header received
0928 06:40:06.054711 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:06.054712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:06.054719 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:06.054775 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:06.054935 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:06.055469 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:06.056383 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:06.056748 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:06.056809 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:06.056819 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:06.056953 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 06:40:06.056989 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:40:06.057076 20310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:06.054424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:45723 (local address 127.18.211.67:34995)
0928 06:40:06.054480 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:06.054491 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:06.054530 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:06.054776 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:06.054776 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:06.054778 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:06.054847 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:06.054849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:06.055516 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:06.055517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:06.056301 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:06.056303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:06.056818 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:06.056819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:06.056896 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:06.056898 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:40:06.056941 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 06:40:06.056978 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":41}
I20260928 06:40:06.057083 20331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:06.054424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:42919 (local address 127.18.211.67:45713)
0928 06:40:06.054612 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:06.054620 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:06.054667 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:06.054793 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:06.054794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:06.054796 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:06.054875 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:06.054876 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:06.055540 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:06.055541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:06.056303 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:06.056305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:06.056826 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:06.056827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:06.056893 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:06.056897 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:40:06.056941 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 06:40:06.056978 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":15,"thread_start_us":114,"threads_started":1}
I20260928 06:40:06.057391 20128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a37e1cc7e6945cf90d08d32962eaeb1" candidate_uuid: "ce17261982124a17b372dc06bba842a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6767e2d81a5146dd94d65223fa008047" is_pre_election: true
I20260928 06:40:06.057542 20128 raft_consensus.cc:2454] T 0a37e1cc7e6945cf90d08d32962eaeb1 P 6767e2d81a5146dd94d65223fa008047 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce17261982124a17b372dc06bba842a7 in term 0.
I20260928 06:40:06.057564 19994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a37e1cc7e6945cf90d08d32962eaeb1" candidate_uuid: "ce17261982124a17b372dc06bba842a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e5e90843a445c8be90cd1aecf655ca" is_pre_election: true
I20260928 06:40:06.057696 19994 raft_consensus.cc:2454] T 0a37e1cc7e6945cf90d08d32962eaeb1 P c5e5e90843a445c8be90cd1aecf655ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce17261982124a17b372dc06bba842a7 in term 0.
I20260928 06:40:06.057785 20196 leader_election.cc:302] T 0a37e1cc7e6945cf90d08d32962eaeb1 P ce17261982124a17b372dc06bba842a7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6767e2d81a5146dd94d65223fa008047, ce17261982124a17b372dc06bba842a7; no voters:
I20260928 06:40:06.057946 20327 raft_consensus.cc:2781] T 0a37e1cc7e6945cf90d08d32962eaeb1 P ce17261982124a17b372dc06bba842a7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:40:06.057991 20327 raft_consensus.cc:482] T 0a37e1cc7e6945cf90d08d32962eaeb1 P ce17261982124a17b372dc06bba842a7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:40:06.058012 20327 raft_consensus.cc:3037] T 0a37e1cc7e6945cf90d08d32962eaeb1 P ce17261982124a17b372dc06bba842a7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:06.058718 20327 raft_consensus.cc:504] T 0a37e1cc7e6945cf90d08d32962eaeb1 P ce17261982124a17b372dc06bba842a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce17261982124a17b372dc06bba842a7" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40111 } } peers { permanent_uuid: "c5e5e90843a445c8be90cd1aecf655ca" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 42919 } } peers { permanent_uuid: "6767e2d81a5146dd94d65223fa008047" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 45723 } }
I20260928 06:40:06.058919 20327 leader_election.cc:288] T 0a37e1cc7e6945cf90d08d32962eaeb1 P ce17261982124a17b372dc06bba842a7 [CANDIDATE]: Term 1 election: Requested vote from peers c5e5e90843a445c8be90cd1aecf655ca (127.18.211.65:42919), 6767e2d81a5146dd94d65223fa008047 (127.18.211.66:45723)
I20260928 06:40:06.059053 19994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a37e1cc7e6945cf90d08d32962eaeb1" candidate_uuid: "ce17261982124a17b372dc06bba842a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5e5e90843a445c8be90cd1aecf655ca"
I20260928 06:40:06.059057 20128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a37e1cc7e6945cf90d08d32962eaeb1" candidate_uuid: "ce17261982124a17b372dc06bba842a7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6767e2d81a5146dd94d65223fa008047"
I20260928 06:40:06.059147 19994 raft_consensus.cc:3037] T 0a37e1cc7e6945cf90d08d32962eaeb1 P c5e5e90843a445c8be90cd1aecf655ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:06.059160 20128 raft_consensus.cc:3037] T 0a37e1cc7e6945cf90d08d32962eaeb1 P 6767e2d81a5146dd94d65223fa008047 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:06.059847 19994 raft_consensus.cc:2454] T 0a37e1cc7e6945cf90d08d32962eaeb1 P c5e5e90843a445c8be90cd1aecf655ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce17261982124a17b372dc06bba842a7 in term 1.
I20260928 06:40:06.059924 20128 raft_consensus.cc:2454] T 0a37e1cc7e6945cf90d08d32962eaeb1 P 6767e2d81a5146dd94d65223fa008047 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce17261982124a17b372dc06bba842a7 in term 1.
I20260928 06:40:06.060088 20196 leader_election.cc:302] T 0a37e1cc7e6945cf90d08d32962eaeb1 P ce17261982124a17b372dc06bba842a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6767e2d81a5146dd94d65223fa008047, ce17261982124a17b372dc06bba842a7; no voters:
I20260928 06:40:06.060186 20327 raft_consensus.cc:2781] T 0a37e1cc7e6945cf90d08d32962eaeb1 P ce17261982124a17b372dc06bba842a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:06.060372 20327 raft_consensus.cc:686] T 0a37e1cc7e6945cf90d08d32962eaeb1 P ce17261982124a17b372dc06bba842a7 [term 1 LEADER]: Becoming Leader. State: Replica: ce17261982124a17b372dc06bba842a7, State: Running, Role: LEADER
I20260928 06:40:06.060463 20327 consensus_queue.cc:237] T 0a37e1cc7e6945cf90d08d32962eaeb1 P ce17261982124a17b372dc06bba842a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce17261982124a17b372dc06bba842a7" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40111 } } peers { permanent_uuid: "c5e5e90843a445c8be90cd1aecf655ca" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 42919 } } peers { permanent_uuid: "6767e2d81a5146dd94d65223fa008047" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 45723 } }
I20260928 06:40:06.061266 19847 catalog_manager.cc:5495] T 0a37e1cc7e6945cf90d08d32962eaeb1 P ce17261982124a17b372dc06bba842a7 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce17261982124a17b372dc06bba842a7 (127.18.211.67). New cstate: current_term: 1 leader_uuid: "ce17261982124a17b372dc06bba842a7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce17261982124a17b372dc06bba842a7" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5e5e90843a445c8be90cd1aecf655ca" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 42919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6767e2d81a5146dd94d65223fa008047" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 45723 } health_report { overall_health: UNKNOWN } } }
W20260928 06:40:06.067327 20175 tablet.cc:2369] T 0a37e1cc7e6945cf90d08d32962eaeb1 P 6767e2d81a5146dd94d65223fa008047: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 06:40:06 dist-test-slave-b28r krb5kdc[19803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.65@KRBTEST.COM
I20260928 06:40:06.071756 20320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:06.064597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55908 (local address 127.18.211.65:42919)
0928 06:40:06.064638 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:06.064640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:06.064763 (+ 123us) server_negotiation.cc:407] Connection header received
0928 06:40:06.064823 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:06.064824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:06.064834 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:06.064879 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:06.065452 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:06.065919 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:06.066607 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:06.066799 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:06.066855 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:06.069115 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:06.069126 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:06.069127 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:06.069134 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:06.070625 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:06.070978 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:06.070983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:06.070983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:06.071038 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:06.071230 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:06.071231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:06.071231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:06.071383 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:06.071508 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:06.071587 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 06:40:06.071663 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 28 06:40:06 dist-test-slave-b28r krb5kdc[19803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577606, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 06:40:06 dist-test-slave-b28r krb5kdc[19803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577606, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.211.65@KRBTEST.COM
I20260928 06:40:06.093225 20320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:06.086719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55910 (local address 127.18.211.65:42919)
0928 06:40:06.086779 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:06.086781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:06.086837 (+ 56us) server_negotiation.cc:407] Connection header received
0928 06:40:06.086902 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:06.086903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:06.086913 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:06.086959 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:06.087564 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:06.088050 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:06.088745 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:06.088933 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:06.088987 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:06.090972 (+ 1985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:06.090973 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:06.090974 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:06.090981 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:06.092199 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:06.092492 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:06.092493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:06.092493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:06.092543 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:06.092704 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:06.092705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:06.092705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:06.092805 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:06.092887 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:06.093054 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 06:40:06.093131 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 06:40:06 dist-test-slave-b28r krb5kdc[19803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:40:06 dist-test-slave-b28r krb5kdc[19803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577606, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.65@KRBTEST.COM
I20260928 06:40:06.114876 20320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:06.108406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55914 (local address 127.18.211.65:42919)
0928 06:40:06.108480 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:06.108483 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:06.108519 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:40:06.108601 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:06.108601 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:06.108611 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:06.108656 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:06.109168 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:06.109635 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:06.110362 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:06.110544 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:06.110597 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:06.112635 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:06.112636 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:06.112636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:06.112644 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:06.113880 (+ 1236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:06.114171 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:06.114172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:06.114173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:06.114222 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:06.114413 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:06.114414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:06.114414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:06.114510 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:06.114591 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:06.114707 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 06:40:06.114782 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 06:40:06.115363 19954 tablet_service.cc:2908] Rejecting scan request for tablet 0a37e1cc7e6945cf90d08d32962eaeb1: Uninitialized: clean time has not yet been initialized
I20260928 06:40:06.116263 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 19911
I20260928 06:40:06.120436 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 20045
I20260928 06:40:06.124534 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 20179
I20260928 06:40:06.128707 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 19816
2026-09-28T06:40:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1823 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:40:06 dist-test-slave-b28r krb5kdc[20346](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:40:06 dist-test-slave-b28r krb5kdc[20346](info): set up 2 sockets
Sep 28 06:40:06 dist-test-slave-b28r krb5kdc[20346](info): commencing operation
krb5kdc: starting...
W20260928 06:40:07.075367 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:40:07 dist-test-slave-b28r krb5kdc[20346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:40:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:40:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.211.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
I20260928 06:40:07.185564 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:35495
--webserver_interface=127.18.211.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:39237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:35495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:07.264850 20359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:07.265048 20359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:07.265080 20359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:07.266184 20359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:40:07.266223 20359 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 06:40:07.266240 20359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:07.266254 20359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:07.266270 20359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:40:07.266294 20359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:40:07.267777 20359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:39237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:35495
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:35495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:07.268028 20359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:07.268235 20359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:07.272970 20367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:07.272969 20365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:07.273061 20364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:07.273095 20359 server_base.cc:1034] running on GCE node
I20260928 06:40:07.273399 20359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:07.273666 20359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:07.274812 20359 hybrid_clock.cc:648] HybridClock initialized: now 1790577607274797 us; error 31 us; skew 500 ppm
Sep 28 06:40:07 dist-test-slave-b28r krb5kdc[20346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577607, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:07.277228 20359 init.cc:377] Logged in from keytab as kudu/127.18.211.126@KRBTEST.COM (short username kudu)
I20260928 06:40:07.277644 20359 webserver.cc:460] Webserver started at http://127.18.211.126:40743/ using document root <none> and password file <none>
I20260928 06:40:07.277865 20359 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:07.277913 20359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:07.278031 20359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:07.279033 20359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "38fcbd2db9a0413da62e6eadc0f9b220"
format_stamp: "Formatted at 2026-09-28 06:40:07 on dist-test-slave-b28r"
server_key: "7b9db4eb594bacc193ff2d22e3a21637"
server_key_iv: "5a9e5a8a08d3dbcf4420c1f69ceb08f6"
server_key_version: "encryptionkey@0"
I20260928 06:40:07.279402 20359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "38fcbd2db9a0413da62e6eadc0f9b220"
format_stamp: "Formatted at 2026-09-28 06:40:07 on dist-test-slave-b28r"
server_key: "7b9db4eb594bacc193ff2d22e3a21637"
server_key_iv: "5a9e5a8a08d3dbcf4420c1f69ceb08f6"
server_key_version: "encryptionkey@0"
I20260928 06:40:07.280647 20359 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:07.281466 20374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:07.281628 20359 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:07.281703 20359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "38fcbd2db9a0413da62e6eadc0f9b220"
format_stamp: "Formatted at 2026-09-28 06:40:07 on dist-test-slave-b28r"
server_key: "7b9db4eb594bacc193ff2d22e3a21637"
server_key_iv: "5a9e5a8a08d3dbcf4420c1f69ceb08f6"
server_key_version: "encryptionkey@0"
I20260928 06:40:07.281781 20359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:07.313009 20359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:07.315016 20359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:07.315181 20359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:07.319456 20359 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:35495
I20260928 06:40:07.319522 20436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:35495 every 8 connection(s)
I20260928 06:40:07.319855 20359 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:40:07.320324 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 20359
I20260928 06:40:07.320405 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:40:07.320554 19277 external_mini_cluster.cc:1428] Setting key 51b79ec1736186ebb9d50708c9883c1d
I20260928 06:40:07.320547 20437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:07.323841 20437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38fcbd2db9a0413da62e6eadc0f9b220: Bootstrap starting.
I20260928 06:40:07.324486 20437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38fcbd2db9a0413da62e6eadc0f9b220: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:07.324762 20437 log.cc:824] T 00000000000000000000000000000000 P 38fcbd2db9a0413da62e6eadc0f9b220: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:07.325515 20437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38fcbd2db9a0413da62e6eadc0f9b220: No bootstrap required, opened a new log
Sep 28 06:40:07 dist-test-slave-b28r krb5kdc[20346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577607, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:07.327446 20437 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38fcbd2db9a0413da62e6eadc0f9b220 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fcbd2db9a0413da62e6eadc0f9b220" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35495 } }
I20260928 06:40:07.327656 20437 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38fcbd2db9a0413da62e6eadc0f9b220 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:07.327703 20437 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38fcbd2db9a0413da62e6eadc0f9b220 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38fcbd2db9a0413da62e6eadc0f9b220, State: Initialized, Role: FOLLOWER
I20260928 06:40:07.327839 20437 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38fcbd2db9a0413da62e6eadc0f9b220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fcbd2db9a0413da62e6eadc0f9b220" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35495 } }
I20260928 06:40:07.327926 20437 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38fcbd2db9a0413da62e6eadc0f9b220 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:40:07.327950 20437 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38fcbd2db9a0413da62e6eadc0f9b220 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:40:07.327998 20437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38fcbd2db9a0413da62e6eadc0f9b220 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:07.328511 20437 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38fcbd2db9a0413da62e6eadc0f9b220 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fcbd2db9a0413da62e6eadc0f9b220" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35495 } }
I20260928 06:40:07.328604 20437 leader_election.cc:302] T 00000000000000000000000000000000 P 38fcbd2db9a0413da62e6eadc0f9b220 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38fcbd2db9a0413da62e6eadc0f9b220; no voters:
I20260928 06:40:07.328755 20437 leader_election.cc:288] T 00000000000000000000000000000000 P 38fcbd2db9a0413da62e6eadc0f9b220 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:40:07.328795 20442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38fcbd2db9a0413da62e6eadc0f9b220 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:07.328917 20442 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38fcbd2db9a0413da62e6eadc0f9b220 [term 1 LEADER]: Becoming Leader. State: Replica: 38fcbd2db9a0413da62e6eadc0f9b220, State: Running, Role: LEADER
I20260928 06:40:07.328992 20442 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38fcbd2db9a0413da62e6eadc0f9b220 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fcbd2db9a0413da62e6eadc0f9b220" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35495 } }
I20260928 06:40:07.329022 20437 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38fcbd2db9a0413da62e6eadc0f9b220 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:40:07.329295 20442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38fcbd2db9a0413da62e6eadc0f9b220 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38fcbd2db9a0413da62e6eadc0f9b220. Latest consensus state: current_term: 1 leader_uuid: "38fcbd2db9a0413da62e6eadc0f9b220" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fcbd2db9a0413da62e6eadc0f9b220" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35495 } } }
I20260928 06:40:07.329365 20442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38fcbd2db9a0413da62e6eadc0f9b220 [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:07.329335 20444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38fcbd2db9a0413da62e6eadc0f9b220 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38fcbd2db9a0413da62e6eadc0f9b220" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38fcbd2db9a0413da62e6eadc0f9b220" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35495 } } }
I20260928 06:40:07.329550 20444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38fcbd2db9a0413da62e6eadc0f9b220 [sys.catalog]: This master's current role is: LEADER
W20260928 06:40:07.330788 20457 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 38fcbd2db9a0413da62e6eadc0f9b220: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:40:07.330955 20457 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:40:07.331067 20447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:40:07.331117 20440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:07.321049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36676 (local address 127.18.211.126:35495)
0928 06:40:07.321280 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:07.321290 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:07.321326 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:40:07.321740 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:07.321750 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:07.321815 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:07.321965 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:07.322619 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.323975 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.324786 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.325284 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.325393 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:07.327533 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:07.327537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:07.327537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:07.327549 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:07.329458 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:07.329856 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:07.329858 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:07.329858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:07.329941 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:07.330174 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:07.330175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:07.330175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:07.330504 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:07.330649 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:07.330718 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 06:40:07.330859 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":130,"threads_started":1}
I20260928 06:40:07.331269 20447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:40:07.332932 20447 catalog_manager.cc:1347] Generated new cluster ID: 247209bfc13e41b9ab4de26a42fded1c
I20260928 06:40:07.332983 20447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:40:07.352833 20447 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:40:07.353327 20447 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:40:07.356936 20447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38fcbd2db9a0413da62e6eadc0f9b220: Generated new TSK 0
I20260928 06:40:07.357100 20447 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.211.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab.
Entry for principal kudu/127.18.211.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab.
I20260928 06:40:07.385334 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.65:0
--local_ip_for_outbound_sockets=127.18.211.65
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35495
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:39237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:07.463044 20463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:07.463263 20463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:07.463294 20463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:07.464382 20463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:07.464416 20463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:07.464435 20463 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:40:07.464450 20463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.65
I20260928 06:40:07.466009 20463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:39237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.211.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.211.126:35495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.65
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:07.466230 20463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:07.466450 20463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:07.471140 20469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:07.471140 20468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:07.471220 20471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:07.471354 20463 server_base.cc:1034] running on GCE node
I20260928 06:40:07.471531 20463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:07.471755 20463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:07.472910 20463 hybrid_clock.cc:648] HybridClock initialized: now 1790577607472894 us; error 27 us; skew 500 ppm
Sep 28 06:40:07 dist-test-slave-b28r krb5kdc[20346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577607, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:07.475340 20463 init.cc:377] Logged in from keytab as kudu/127.18.211.65@KRBTEST.COM (short username kudu)
I20260928 06:40:07.475741 20463 webserver.cc:460] Webserver started at http://127.18.211.65:36295/ using document root <none> and password file <none>
I20260928 06:40:07.475932 20463 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:07.475978 20463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:07.476097 20463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:07.477087 20463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-0/data/instance:
uuid: "2d71c58ff2f040cdbde917b51cb265a5"
format_stamp: "Formatted at 2026-09-28 06:40:07 on dist-test-slave-b28r"
server_key: "211be4e4e66d6581ea1663a380e1c146"
server_key_iv: "3196ff507f25a804b46e735b1cc4f6ab"
server_key_version: "encryptionkey@0"
I20260928 06:40:07.477407 20463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance:
uuid: "2d71c58ff2f040cdbde917b51cb265a5"
format_stamp: "Formatted at 2026-09-28 06:40:07 on dist-test-slave-b28r"
server_key: "211be4e4e66d6581ea1663a380e1c146"
server_key_iv: "3196ff507f25a804b46e735b1cc4f6ab"
server_key_version: "encryptionkey@0"
I20260928 06:40:07.478663 20463 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:40:07.479511 20478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:07.479655 20463 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:07.479738 20463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-0/wal
uuid: "2d71c58ff2f040cdbde917b51cb265a5"
format_stamp: "Formatted at 2026-09-28 06:40:07 on dist-test-slave-b28r"
server_key: "211be4e4e66d6581ea1663a380e1c146"
server_key_iv: "3196ff507f25a804b46e735b1cc4f6ab"
server_key_version: "encryptionkey@0"
I20260928 06:40:07.479801 20463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:07.498303 20463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:07.500337 20463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:07.500473 20463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:07.500974 20463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:07.501014 20463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:07.501041 20463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:07.501058 20463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:40:07 dist-test-slave-b28r krb5kdc[20346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577607, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:07.507520 20463 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.65:35019
I20260928 06:40:07.507634 20593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.65:35019 every 8 connection(s)
I20260928 06:40:07.507946 20463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
I20260928 06:40:07.509536 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 20463
I20260928 06:40:07.509629 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance
I20260928 06:40:07.509805 19277 external_mini_cluster.cc:1428] Setting key 0b31cececc474fabc03c4989aacbeb6c
I20260928 06:40:07.510144 20440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:07.501407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:54931 (local address 127.18.211.126:35495)
0928 06:40:07.501475 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:07.501477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:07.501619 (+ 142us) server_negotiation.cc:407] Connection header received
0928 06:40:07.502071 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:07.502072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:07.502085 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:07.502158 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:07.502922 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.503535 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.504577 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.504825 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.504890 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:07.507506 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:07.507507 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:07.507508 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:07.507515 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:07.508918 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:07.509255 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:07.509255 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:07.509256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:07.509301 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:07.509511 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:07.509511 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:07.509511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:07.509604 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:07.509658 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:07.509971 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 06:40:07.510055 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 06:40:07.510298 20487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:07.501338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35495 (local address 127.18.211.65:54931)
0928 06:40:07.501498 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:07.501537 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:07.501985 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:07.502228 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:07.502232 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:07.502555 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:07.502834 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:07.502841 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.503622 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.503624 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:07.504483 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:07.504486 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.504907 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.504908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:07.505015 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:07.505468 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:07.505483 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:07.507414 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:07.508988 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:07.508992 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:07.508994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:07.509196 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:07.509335 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:07.509336 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:07.509336 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:07.509455 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:07.509702 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:07.509709 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:07.509826 (+ 117us) client_negotiation.cc:769] Sending connection context
0928 06:40:07.509947 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 06:40:07.510099 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
I20260928 06:40:07.511861 20594 heartbeater.cc:347] Connected to a master server at 127.18.211.126:35495
I20260928 06:40:07.511945 20594 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:07.512122 20594 heartbeater.cc:511] Master 127.18.211.126:35495 requested a full tablet report, sending...
I20260928 06:40:07.512687 20390 ts_manager.cc:194] Registered new tserver with Master: 2d71c58ff2f040cdbde917b51cb265a5 (127.18.211.65:35019)
I20260928 06:40:07.513490 20390 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.65@KRBTEST.COM'} at 127.18.211.65:54931
WARNING: no policy specified for kudu/127.18.211.66@KRBTEST.COM; defaulting to no policy
I20260928 06:40:07.518627 20440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:07.512911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:55491 (local address 127.18.211.126:35495)
0928 06:40:07.512945 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:07.512947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:07.513082 (+ 135us) server_negotiation.cc:407] Connection header received
0928 06:40:07.513207 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:07.513207 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:07.513217 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:07.513274 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:07.513630 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.514340 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.515354 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.515535 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.515590 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:07.516464 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:07.516465 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:07.516466 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:07.516472 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:07.517669 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:07.517961 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:07.517962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:07.517963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:07.518011 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:07.518169 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:07.518170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:07.518170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:07.518265 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:07.518319 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:07.518508 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 06:40:07.518568 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:40:07.518639 20601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:07.512881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35495 (local address 127.18.211.65:55491)
0928 06:40:07.513031 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:07.513041 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:07.513091 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:07.513284 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:07.513285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:07.513470 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:07.513529 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:07.513530 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.514456 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.514457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:07.515277 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:07.515280 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.515603 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.515604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:07.515676 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:07.516020 (+ 344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:07.516025 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:07.516369 (+ 344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:07.517739 (+ 1370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:07.517739 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:07.517740 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:07.517902 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:07.518062 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:07.518063 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:07.518063 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:07.518121 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:07.518320 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:07.518321 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:07.518403 (+ 82us) client_negotiation.cc:769] Sending connection context
0928 06:40:07.518467 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 06:40:07.518565 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":59,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab.
Entry for principal kudu/127.18.211.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab.
I20260928 06:40:07.531132 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.66:0
--local_ip_for_outbound_sockets=127.18.211.66
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35495
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:39237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:07.610389 20603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:07.610580 20603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:07.610610 20603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:07.611804 20603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:07.611845 20603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:07.611871 20603 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:40:07.611896 20603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.66
I20260928 06:40:07.613315 20603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:39237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.211.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.211.126:35495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.66
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:07.613545 20603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:07.613744 20603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:07.618474 20609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:07.618498 20608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:07.618554 20603 server_base.cc:1034] running on GCE node
W20260928 06:40:07.618497 20611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:07.618865 20603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:07.619122 20603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:07.620260 20603 hybrid_clock.cc:648] HybridClock initialized: now 1790577607620246 us; error 33 us; skew 500 ppm
Sep 28 06:40:07 dist-test-slave-b28r krb5kdc[20346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577607, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:07.622581 20603 init.cc:377] Logged in from keytab as kudu/127.18.211.66@KRBTEST.COM (short username kudu)
I20260928 06:40:07.622962 20603 webserver.cc:460] Webserver started at http://127.18.211.66:45123/ using document root <none> and password file <none>
I20260928 06:40:07.623205 20603 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:07.623250 20603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:07.623344 20603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:07.624313 20603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-1/data/instance:
uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3"
format_stamp: "Formatted at 2026-09-28 06:40:07 on dist-test-slave-b28r"
server_key: "5ce10f40e788d49331d28dd419583c74"
server_key_iv: "b689c12bdc34b235ffdfe3bbb6a8d3ad"
server_key_version: "encryptionkey@0"
I20260928 06:40:07.624609 20603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance:
uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3"
format_stamp: "Formatted at 2026-09-28 06:40:07 on dist-test-slave-b28r"
server_key: "5ce10f40e788d49331d28dd419583c74"
server_key_iv: "b689c12bdc34b235ffdfe3bbb6a8d3ad"
server_key_version: "encryptionkey@0"
I20260928 06:40:07.625779 20603 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:40:07.626430 20618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:07.626567 20603 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:40:07.626632 20603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-1/wal
uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3"
format_stamp: "Formatted at 2026-09-28 06:40:07 on dist-test-slave-b28r"
server_key: "5ce10f40e788d49331d28dd419583c74"
server_key_iv: "b689c12bdc34b235ffdfe3bbb6a8d3ad"
server_key_version: "encryptionkey@0"
I20260928 06:40:07.626694 20603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:07.635350 20603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:07.636951 20603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:07.637071 20603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:07.637532 20603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:07.637568 20603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:07.637588 20603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:07.637599 20603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:40:07 dist-test-slave-b28r krb5kdc[20346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577607, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:07.644141 20603 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.66:34597
I20260928 06:40:07.644203 20733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.66:34597 every 8 connection(s)
I20260928 06:40:07.644485 20603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
I20260928 06:40:07.644958 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 20603
I20260928 06:40:07.645041 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance
I20260928 06:40:07.645222 19277 external_mini_cluster.cc:1428] Setting key 76cb256acda2feb91bf8a7fe3372165e
I20260928 06:40:07.647028 20440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:07.637938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:47209 (local address 127.18.211.126:35495)
0928 06:40:07.638001 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:07.638003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:07.638160 (+ 157us) server_negotiation.cc:407] Connection header received
0928 06:40:07.638672 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:07.638673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:07.638685 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:07.638735 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:07.639541 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.640280 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.641319 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.641614 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.641680 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:07.644300 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:07.644301 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:07.644302 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:07.644310 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:07.645739 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:07.646180 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:07.646181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:07.646182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:07.646243 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:07.646455 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:07.646456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:07.646456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:07.646592 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:07.646652 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:07.646863 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 06:40:07.646940 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 06:40:07.647213 20627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:07.637890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35495 (local address 127.18.211.66:47209)
0928 06:40:07.638040 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:07.638080 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:07.638571 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:07.638806 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:07.638810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:07.639168 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:07.639429 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:07.639436 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.640375 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.640376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:07.641233 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:07.641236 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.641694 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.641695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:07.641846 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:07.642376 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:07.642385 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:07.644203 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:07.645817 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:07.645825 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:07.645827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:07.646101 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:07.646307 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:07.646307 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:07.646308 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:07.646405 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:07.646644 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:07.646646 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:07.646756 (+ 110us) client_negotiation.cc:769] Sending connection context
0928 06:40:07.646832 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 06:40:07.646978 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260928 06:40:07.647708 20734 heartbeater.cc:347] Connected to a master server at 127.18.211.126:35495
I20260928 06:40:07.647794 20734 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:07.647984 20734 heartbeater.cc:511] Master 127.18.211.126:35495 requested a full tablet report, sending...
I20260928 06:40:07.648366 20391 ts_manager.cc:194] Registered new tserver with Master: b1e75abdf38c4d00a00b7d8a0f506eb3 (127.18.211.66:34597)
I20260928 06:40:07.648792 20391 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.66@KRBTEST.COM'} at 127.18.211.66:47209
WARNING: no policy specified for kudu/127.18.211.67@KRBTEST.COM; defaulting to no policy
I20260928 06:40:07.654345 20741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:07.648692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35495 (local address 127.18.211.66:40195)
0928 06:40:07.648822 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:07.648832 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:07.648905 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:07.649017 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:07.649017 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:07.649211 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:07.649267 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:07.649269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.650036 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.650037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:07.650704 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:07.650706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.651192 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.651192 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:07.651245 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:07.651621 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:07.651626 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:07.651902 (+ 276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:07.653406 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:07.653407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:07.653407 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:07.653558 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:07.653748 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:07.653748 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:07.653749 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:07.653799 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:07.654100 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:07.654101 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:07.654145 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 06:40:07.654199 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 06:40:07.654278 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260928 06:40:07.654381 20440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:07.648755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:40195 (local address 127.18.211.126:35495)
0928 06:40:07.648789 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:07.648791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:07.648880 (+ 89us) server_negotiation.cc:407] Connection header received
0928 06:40:07.648952 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:07.648953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:07.648963 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:07.649013 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:07.649339 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.649940 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.650779 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.651117 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.651194 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:07.651980 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:07.651981 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:07.651981 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:07.651988 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:07.653331 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:07.653627 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:07.653628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:07.653629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:07.653687 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:07.653846 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:07.653847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:07.653847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:07.654038 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:07.654113 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:07.654229 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 06:40:07.654299 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab.
Entry for principal kudu/127.18.211.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab.
I20260928 06:40:07.669112 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.67:0
--local_ip_for_outbound_sockets=127.18.211.67
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35495
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:39237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:07.747225 20743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:07.747483 20743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:07.747520 20743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:07.748764 20743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:07.748809 20743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:07.748836 20743 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:40:07.748860 20743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.67
I20260928 06:40:07.750347 20743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:39237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.211.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.211.126:35495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.67
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:07.750594 20743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:07.750813 20743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:07.755465 20749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:07.755496 20751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:07.755931 20748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:07.755960 20743 server_base.cc:1034] running on GCE node
I20260928 06:40:07.756188 20743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:07.756417 20743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:07.757557 20743 hybrid_clock.cc:648] HybridClock initialized: now 1790577607757545 us; error 31 us; skew 500 ppm
Sep 28 06:40:07 dist-test-slave-b28r krb5kdc[20346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577607, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:07.759996 20743 init.cc:377] Logged in from keytab as kudu/127.18.211.67@KRBTEST.COM (short username kudu)
I20260928 06:40:07.760551 20743 webserver.cc:460] Webserver started at http://127.18.211.67:38071/ using document root <none> and password file <none>
I20260928 06:40:07.760752 20743 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:07.760800 20743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:07.760917 20743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:07.761921 20743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-2/data/instance:
uuid: "91c3a3280e414ab4a758e8cdedb881c9"
format_stamp: "Formatted at 2026-09-28 06:40:07 on dist-test-slave-b28r"
server_key: "8e81809acb419adc6b72507c7d35473a"
server_key_iv: "545ed77dbbac175b2c9e4cc1095f3fb0"
server_key_version: "encryptionkey@0"
I20260928 06:40:07.762236 20743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance:
uuid: "91c3a3280e414ab4a758e8cdedb881c9"
format_stamp: "Formatted at 2026-09-28 06:40:07 on dist-test-slave-b28r"
server_key: "8e81809acb419adc6b72507c7d35473a"
server_key_iv: "545ed77dbbac175b2c9e4cc1095f3fb0"
server_key_version: "encryptionkey@0"
I20260928 06:40:07.763526 20743 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:40:07.764221 20758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:07.764405 20743 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:40:07.764487 20743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-2/wal
uuid: "91c3a3280e414ab4a758e8cdedb881c9"
format_stamp: "Formatted at 2026-09-28 06:40:07 on dist-test-slave-b28r"
server_key: "8e81809acb419adc6b72507c7d35473a"
server_key_iv: "545ed77dbbac175b2c9e4cc1095f3fb0"
server_key_version: "encryptionkey@0"
I20260928 06:40:07.764542 20743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:07.773010 20743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:07.775017 20743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:07.775174 20743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:07.775655 20743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:07.775693 20743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:07.775713 20743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:07.775724 20743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:07.781814 20743 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.67:40927
Sep 28 06:40:07 dist-test-slave-b28r krb5kdc[20346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577607, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:07.781965 20873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.67:40927 every 8 connection(s)
I20260928 06:40:07.782179 20743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
I20260928 06:40:07.783329 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 20743
I20260928 06:40:07.783406 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance
I20260928 06:40:07.783569 19277 external_mini_cluster.cc:1428] Setting key a4abaab0e16bb0f641587a56571f6d10
I20260928 06:40:07.785174 20440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:07.776080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:60039 (local address 127.18.211.126:35495)
0928 06:40:07.776144 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:07.776146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:07.776352 (+ 206us) server_negotiation.cc:407] Connection header received
0928 06:40:07.776829 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:07.776830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:07.776842 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:07.776919 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:07.777718 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.778461 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.779930 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.780226 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.780298 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:07.782529 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:07.782530 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:07.782530 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:07.782538 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:07.784090 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:07.784502 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:07.784503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:07.784504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:07.784558 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:07.784724 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:07.784725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:07.784725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:07.784822 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:07.784866 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:07.785052 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 06:40:07.785111 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:40:07.785354 20767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:07.776043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35495 (local address 127.18.211.67:60039)
0928 06:40:07.776223 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:07.776264 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:07.776707 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:07.776997 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:07.777001 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:07.777351 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:07.777616 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:07.777623 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.778546 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.778546 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:07.779837 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:07.779840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.780310 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.780311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:07.780411 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:07.780881 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:07.780889 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:07.782425 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:07.784163 (+ 1738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:07.784169 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:07.784171 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:07.784445 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:07.784598 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:07.784599 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:07.784599 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:07.784686 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:07.784857 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:07.784860 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:07.784961 (+ 101us) client_negotiation.cc:769] Sending connection context
0928 06:40:07.785035 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 06:40:07.785152 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":70,"threads_started":1}
I20260928 06:40:07.785780 20874 heartbeater.cc:347] Connected to a master server at 127.18.211.126:35495
I20260928 06:40:07.785857 20874 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:07.786017 20874 heartbeater.cc:511] Master 127.18.211.126:35495 requested a full tablet report, sending...
I20260928 06:40:07.786365 20390 ts_manager.cc:194] Registered new tserver with Master: 91c3a3280e414ab4a758e8cdedb881c9 (127.18.211.67:40927)
I20260928 06:40:07.786724 20390 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.67@KRBTEST.COM'} at 127.18.211.67:60039
I20260928 06:40:07.791152 20440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:07.786738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:41267 (local address 127.18.211.126:35495)
0928 06:40:07.786777 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:07.786778 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:07.786897 (+ 119us) server_negotiation.cc:407] Connection header received
0928 06:40:07.786939 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:07.786939 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:07.786947 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:07.786985 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:07.787330 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.787761 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.788382 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.788549 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.788592 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:07.789261 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:07.789262 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:07.789262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:07.789267 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:07.790415 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:07.790664 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:07.790664 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:07.790665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:07.790706 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:07.790816 (+ 110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:07.790817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:07.790817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:07.790896 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:07.790937 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:07.791025 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 06:40:07.791071 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:40:07.791200 20880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:07.786679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35495 (local address 127.18.211.67:41267)
0928 06:40:07.786830 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:07.786840 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:07.786896 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:07.786989 (+ 93us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:07.786990 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:07.787214 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:07.787268 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:07.787270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.787835 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.787836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:07.788324 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:07.788326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.788595 (+ 269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.788595 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:07.788644 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:07.788966 (+ 322us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:07.788969 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:07.789201 (+ 232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:07.790474 (+ 1273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:07.790475 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:07.790475 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:07.790617 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:07.790745 (+ 128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:07.790745 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:07.790745 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:07.790786 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:07.790936 (+ 150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:07.790936 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:07.790967 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 06:40:07.791010 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 06:40:07.791076 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":70,"threads_started":1}
I20260928 06:40:07.795106 19277 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 06:40:07 dist-test-slave-b28r krb5kdc[20346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577607, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 06:40:07 dist-test-slave-b28r krb5kdc[20346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577607, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:07.815626 20440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:07.809252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36678 (local address 127.18.211.126:35495)
0928 06:40:07.809337 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:07.809339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:07.809348 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:40:07.809370 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:07.809371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:07.809379 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:07.809427 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:07.809898 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.810391 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.811160 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.811360 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.811410 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:07.813433 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:07.813434 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:07.813435 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:07.813442 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:07.814607 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:07.814929 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:07.814930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:07.814930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:07.814995 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:07.815173 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:07.815173 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:07.815174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:07.815271 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:07.815363 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:07.815469 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 06:40:07.815535 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 06:40:07.817116 20390 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:40:07.818603 20390 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:40:07.824683 20889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:07.820638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:40927 (local address 127.0.0.1:46662)
0928 06:40:07.821012 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:07.821020 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:07.821064 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:07.821264 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:07.821267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:07.821269 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:07.821369 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:07.821371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.822645 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.822646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:07.823768 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:07.823770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.824405 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.824406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:07.824481 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:07.824494 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 06:40:07.824557 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 06:40:07.824606 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":37,"thread_start_us":65,"threads_started":1}
I20260928 06:40:07.824941 20890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:07.820733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49572 (local address 127.18.211.65:35019)
0928 06:40:07.821366 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:07.821371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:07.821380 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:40:07.821416 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:07.821418 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:07.821434 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:07.821645 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:07.821697 (+ 52us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.822356 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.823769 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.824476 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.824531 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:07.824578 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:07.824658 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 06:40:07.824698 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":87,"threads_started":1}
I20260928 06:40:07.824961 20893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:07.820954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56828 (local address 127.18.211.66:34597)
0928 06:40:07.821094 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:07.821099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:07.821184 (+ 85us) server_negotiation.cc:407] Connection header received
0928 06:40:07.821280 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:07.821282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:07.821297 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:07.821398 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:07.821578 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.822633 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.823695 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.824414 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.824482 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:07.824540 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:07.824840 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 06:40:07.824877 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20260928 06:40:07.824965 20891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:07.820798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46662 (local address 127.18.211.67:40927)
0928 06:40:07.820972 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:07.820977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:07.821113 (+ 136us) server_negotiation.cc:407] Connection header received
0928 06:40:07.821157 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:07.821159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:07.821173 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:07.821247 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:07.821451 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.822570 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.823824 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.824329 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.824392 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:07.824451 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:07.824850 (+ 399us) server_negotiation.cc:299] Negotiation successful
0928 06:40:07.824892 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":74,"threads_started":1}
I20260928 06:40:07.824797 20888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:07.820627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:35019 (local address 127.0.0.1:49572)
0928 06:40:07.820919 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:07.820963 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:07.821030 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:07.821526 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:07.821527 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:07.821529 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:07.821573 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:07.821575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.822433 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.822434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:07.823696 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:07.823699 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.824542 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.824543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:07.824598 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:07.824601 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:40:07.824647 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 06:40:07.824677 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":25,"thread_start_us":58,"threads_started":1}
I20260928 06:40:07.824851 20892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:07.820830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:34597 (local address 127.0.0.1:56828)
0928 06:40:07.821137 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:07.821144 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:07.821189 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:07.821464 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:07.821465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:07.821467 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:07.821505 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:07.821507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.822690 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.822691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:07.823623 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:07.823625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.824576 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.824577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:07.824638 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:07.824642 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:40:07.824686 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 06:40:07.824725 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":1,"thread_start_us":77,"threads_started":1}
I20260928 06:40:07.825728 20528 tablet_service.cc:1467] Processing CreateTablet for tablet 3d276b83b5df4001a4b6c875f2ca96a8 (DEFAULT_TABLE table=test-table [id=cab1989f0bff4928b4ed1c108cab8b4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:07.825742 20808 tablet_service.cc:1467] Processing CreateTablet for tablet 3d276b83b5df4001a4b6c875f2ca96a8 (DEFAULT_TABLE table=test-table [id=cab1989f0bff4928b4ed1c108cab8b4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:07.826043 20528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d276b83b5df4001a4b6c875f2ca96a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:07.826069 20808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d276b83b5df4001a4b6c875f2ca96a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:07.828581 20894 tablet_bootstrap.cc:492] T 3d276b83b5df4001a4b6c875f2ca96a8 P 91c3a3280e414ab4a758e8cdedb881c9: Bootstrap starting.
I20260928 06:40:07.828742 20895 tablet_bootstrap.cc:492] T 3d276b83b5df4001a4b6c875f2ca96a8 P 2d71c58ff2f040cdbde917b51cb265a5: Bootstrap starting.
I20260928 06:40:07.829254 20668 tablet_service.cc:1467] Processing CreateTablet for tablet 3d276b83b5df4001a4b6c875f2ca96a8 (DEFAULT_TABLE table=test-table [id=cab1989f0bff4928b4ed1c108cab8b4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:07.829454 20894 tablet_bootstrap.cc:654] T 3d276b83b5df4001a4b6c875f2ca96a8 P 91c3a3280e414ab4a758e8cdedb881c9: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:07.829483 20895 tablet_bootstrap.cc:654] T 3d276b83b5df4001a4b6c875f2ca96a8 P 2d71c58ff2f040cdbde917b51cb265a5: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:07.829625 20668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d276b83b5df4001a4b6c875f2ca96a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:07.829782 20895 log.cc:824] T 3d276b83b5df4001a4b6c875f2ca96a8 P 2d71c58ff2f040cdbde917b51cb265a5: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:07.829856 20894 log.cc:824] T 3d276b83b5df4001a4b6c875f2ca96a8 P 91c3a3280e414ab4a758e8cdedb881c9: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:07.830595 20894 tablet_bootstrap.cc:492] T 3d276b83b5df4001a4b6c875f2ca96a8 P 91c3a3280e414ab4a758e8cdedb881c9: No bootstrap required, opened a new log
I20260928 06:40:07.830658 20894 ts_tablet_manager.cc:1397] T 3d276b83b5df4001a4b6c875f2ca96a8 P 91c3a3280e414ab4a758e8cdedb881c9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:40:07.832181 20894 raft_consensus.cc:348] T 3d276b83b5df4001a4b6c875f2ca96a8 P 91c3a3280e414ab4a758e8cdedb881c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } }
I20260928 06:40:07.832324 20894 raft_consensus.cc:374] T 3d276b83b5df4001a4b6c875f2ca96a8 P 91c3a3280e414ab4a758e8cdedb881c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:07.832357 20894 raft_consensus.cc:729] T 3d276b83b5df4001a4b6c875f2ca96a8 P 91c3a3280e414ab4a758e8cdedb881c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91c3a3280e414ab4a758e8cdedb881c9, State: Initialized, Role: FOLLOWER
I20260928 06:40:07.832469 20894 consensus_queue.cc:260] T 3d276b83b5df4001a4b6c875f2ca96a8 P 91c3a3280e414ab4a758e8cdedb881c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } }
I20260928 06:40:07.832651 20898 tablet_bootstrap.cc:492] T 3d276b83b5df4001a4b6c875f2ca96a8 P b1e75abdf38c4d00a00b7d8a0f506eb3: Bootstrap starting.
I20260928 06:40:07.832723 20894 ts_tablet_manager.cc:1428] T 3d276b83b5df4001a4b6c875f2ca96a8 P 91c3a3280e414ab4a758e8cdedb881c9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:40:07.832839 20874 heartbeater.cc:503] Master 127.18.211.126:35495 was elected leader, sending a full tablet report...
I20260928 06:40:07.833164 20895 tablet_bootstrap.cc:492] T 3d276b83b5df4001a4b6c875f2ca96a8 P 2d71c58ff2f040cdbde917b51cb265a5: No bootstrap required, opened a new log
I20260928 06:40:07.833249 20895 ts_tablet_manager.cc:1397] T 3d276b83b5df4001a4b6c875f2ca96a8 P 2d71c58ff2f040cdbde917b51cb265a5: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260928 06:40:07.833249 20898 tablet_bootstrap.cc:654] T 3d276b83b5df4001a4b6c875f2ca96a8 P b1e75abdf38c4d00a00b7d8a0f506eb3: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:07.833526 20898 log.cc:824] T 3d276b83b5df4001a4b6c875f2ca96a8 P b1e75abdf38c4d00a00b7d8a0f506eb3: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:07.834189 20898 tablet_bootstrap.cc:492] T 3d276b83b5df4001a4b6c875f2ca96a8 P b1e75abdf38c4d00a00b7d8a0f506eb3: No bootstrap required, opened a new log
I20260928 06:40:07.834240 20898 ts_tablet_manager.cc:1397] T 3d276b83b5df4001a4b6c875f2ca96a8 P b1e75abdf38c4d00a00b7d8a0f506eb3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:40:07.834854 20895 raft_consensus.cc:348] T 3d276b83b5df4001a4b6c875f2ca96a8 P 2d71c58ff2f040cdbde917b51cb265a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } }
I20260928 06:40:07.834980 20895 raft_consensus.cc:374] T 3d276b83b5df4001a4b6c875f2ca96a8 P 2d71c58ff2f040cdbde917b51cb265a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:07.835011 20895 raft_consensus.cc:729] T 3d276b83b5df4001a4b6c875f2ca96a8 P 2d71c58ff2f040cdbde917b51cb265a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d71c58ff2f040cdbde917b51cb265a5, State: Initialized, Role: FOLLOWER
I20260928 06:40:07.835145 20895 consensus_queue.cc:260] T 3d276b83b5df4001a4b6c875f2ca96a8 P 2d71c58ff2f040cdbde917b51cb265a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } }
I20260928 06:40:07.835402 20895 ts_tablet_manager.cc:1428] T 3d276b83b5df4001a4b6c875f2ca96a8 P 2d71c58ff2f040cdbde917b51cb265a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:40:07.835448 20594 heartbeater.cc:503] Master 127.18.211.126:35495 was elected leader, sending a full tablet report...
I20260928 06:40:07.835892 20898 raft_consensus.cc:348] T 3d276b83b5df4001a4b6c875f2ca96a8 P b1e75abdf38c4d00a00b7d8a0f506eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } }
I20260928 06:40:07.836004 20898 raft_consensus.cc:374] T 3d276b83b5df4001a4b6c875f2ca96a8 P b1e75abdf38c4d00a00b7d8a0f506eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:07.836058 20898 raft_consensus.cc:729] T 3d276b83b5df4001a4b6c875f2ca96a8 P b1e75abdf38c4d00a00b7d8a0f506eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1e75abdf38c4d00a00b7d8a0f506eb3, State: Initialized, Role: FOLLOWER
I20260928 06:40:07.836191 20898 consensus_queue.cc:260] T 3d276b83b5df4001a4b6c875f2ca96a8 P b1e75abdf38c4d00a00b7d8a0f506eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } }
I20260928 06:40:07.836437 20898 ts_tablet_manager.cc:1428] T 3d276b83b5df4001a4b6c875f2ca96a8 P b1e75abdf38c4d00a00b7d8a0f506eb3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:40:07.836470 20734 heartbeater.cc:503] Master 127.18.211.126:35495 was elected leader, sending a full tablet report...
I20260928 06:40:07.864820 20899 raft_consensus.cc:482] T 3d276b83b5df4001a4b6c875f2ca96a8 P 91c3a3280e414ab4a758e8cdedb881c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:40:07.864943 20899 raft_consensus.cc:504] T 3d276b83b5df4001a4b6c875f2ca96a8 P 91c3a3280e414ab4a758e8cdedb881c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } }
I20260928 06:40:07.865279 20899 leader_election.cc:288] T 3d276b83b5df4001a4b6c875f2ca96a8 P 91c3a3280e414ab4a758e8cdedb881c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d71c58ff2f040cdbde917b51cb265a5 (127.18.211.65:35019), b1e75abdf38c4d00a00b7d8a0f506eb3 (127.18.211.66:34597)
I20260928 06:40:07.868942 20890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:07.865534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:35345 (local address 127.18.211.65:35019)
0928 06:40:07.865578 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:07.865579 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:07.865604 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:40:07.865656 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:07.865657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:07.865666 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:07.865708 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:07.865924 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.866592 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.868060 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.868608 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.868681 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:07.868690 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:07.868838 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 06:40:07.868871 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:40:07.868964 20767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:07.865476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:35019 (local address 127.18.211.67:35345)
0928 06:40:07.865540 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:07.865553 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:07.865600 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:07.865765 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:07.865766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:07.865767 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:07.865854 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:07.865856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.866687 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.866688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:07.867962 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:07.867965 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.868680 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.868681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:07.868767 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:07.868771 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:40:07.868828 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 06:40:07.868883 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":12}
I20260928 06:40:07.869014 20893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:07.865695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:40543 (local address 127.18.211.66:34597)
0928 06:40:07.865739 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:07.865740 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:07.865842 (+ 102us) server_negotiation.cc:407] Connection header received
0928 06:40:07.865942 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:07.865943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:07.865952 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:07.865975 (+ 23us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:07.866148 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.866830 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.868147 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.868704 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.868763 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:07.868771 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:07.868904 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 06:40:07.868953 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:40:07.869203 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:07.865579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:34597 (local address 127.18.211.67:40543)
0928 06:40:07.865779 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:07.865788 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:07.865870 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:07.866027 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:07.866027 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:07.866029 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:07.866084 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:07.866086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.866892 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.866893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:07.868084 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:07.868086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.868775 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.868775 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:07.868845 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:07.868848 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:40:07.868893 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 06:40:07.868925 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":102,"threads_started":1}
I20260928 06:40:07.869349 20548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d276b83b5df4001a4b6c875f2ca96a8" candidate_uuid: "91c3a3280e414ab4a758e8cdedb881c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d71c58ff2f040cdbde917b51cb265a5" is_pre_election: true
I20260928 06:40:07.869518 20548 raft_consensus.cc:2454] T 3d276b83b5df4001a4b6c875f2ca96a8 P 2d71c58ff2f040cdbde917b51cb265a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91c3a3280e414ab4a758e8cdedb881c9 in term 0.
I20260928 06:40:07.869534 20688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d276b83b5df4001a4b6c875f2ca96a8" candidate_uuid: "91c3a3280e414ab4a758e8cdedb881c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" is_pre_election: true
I20260928 06:40:07.869691 20688 raft_consensus.cc:2454] T 3d276b83b5df4001a4b6c875f2ca96a8 P b1e75abdf38c4d00a00b7d8a0f506eb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91c3a3280e414ab4a758e8cdedb881c9 in term 0.
I20260928 06:40:07.869753 20759 leader_election.cc:302] T 3d276b83b5df4001a4b6c875f2ca96a8 P 91c3a3280e414ab4a758e8cdedb881c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d71c58ff2f040cdbde917b51cb265a5, 91c3a3280e414ab4a758e8cdedb881c9; no voters:
I20260928 06:40:07.869879 20899 raft_consensus.cc:2781] T 3d276b83b5df4001a4b6c875f2ca96a8 P 91c3a3280e414ab4a758e8cdedb881c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:40:07.869937 20899 raft_consensus.cc:482] T 3d276b83b5df4001a4b6c875f2ca96a8 P 91c3a3280e414ab4a758e8cdedb881c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:40:07.869961 20899 raft_consensus.cc:3037] T 3d276b83b5df4001a4b6c875f2ca96a8 P 91c3a3280e414ab4a758e8cdedb881c9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:07.870635 20899 raft_consensus.cc:504] T 3d276b83b5df4001a4b6c875f2ca96a8 P 91c3a3280e414ab4a758e8cdedb881c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } }
I20260928 06:40:07.870751 20899 leader_election.cc:288] T 3d276b83b5df4001a4b6c875f2ca96a8 P 91c3a3280e414ab4a758e8cdedb881c9 [CANDIDATE]: Term 1 election: Requested vote from peers 2d71c58ff2f040cdbde917b51cb265a5 (127.18.211.65:35019), b1e75abdf38c4d00a00b7d8a0f506eb3 (127.18.211.66:34597)
I20260928 06:40:07.870890 20548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d276b83b5df4001a4b6c875f2ca96a8" candidate_uuid: "91c3a3280e414ab4a758e8cdedb881c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d71c58ff2f040cdbde917b51cb265a5"
I20260928 06:40:07.870920 20688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d276b83b5df4001a4b6c875f2ca96a8" candidate_uuid: "91c3a3280e414ab4a758e8cdedb881c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3"
I20260928 06:40:07.870965 20548 raft_consensus.cc:3037] T 3d276b83b5df4001a4b6c875f2ca96a8 P 2d71c58ff2f040cdbde917b51cb265a5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:07.870983 20688 raft_consensus.cc:3037] T 3d276b83b5df4001a4b6c875f2ca96a8 P b1e75abdf38c4d00a00b7d8a0f506eb3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:07.871680 20548 raft_consensus.cc:2454] T 3d276b83b5df4001a4b6c875f2ca96a8 P 2d71c58ff2f040cdbde917b51cb265a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91c3a3280e414ab4a758e8cdedb881c9 in term 1.
I20260928 06:40:07.871701 20688 raft_consensus.cc:2454] T 3d276b83b5df4001a4b6c875f2ca96a8 P b1e75abdf38c4d00a00b7d8a0f506eb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91c3a3280e414ab4a758e8cdedb881c9 in term 1.
I20260928 06:40:07.871819 20759 leader_election.cc:302] T 3d276b83b5df4001a4b6c875f2ca96a8 P 91c3a3280e414ab4a758e8cdedb881c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d71c58ff2f040cdbde917b51cb265a5, 91c3a3280e414ab4a758e8cdedb881c9; no voters:
I20260928 06:40:07.871903 20899 raft_consensus.cc:2781] T 3d276b83b5df4001a4b6c875f2ca96a8 P 91c3a3280e414ab4a758e8cdedb881c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:07.872030 20899 raft_consensus.cc:686] T 3d276b83b5df4001a4b6c875f2ca96a8 P 91c3a3280e414ab4a758e8cdedb881c9 [term 1 LEADER]: Becoming Leader. State: Replica: 91c3a3280e414ab4a758e8cdedb881c9, State: Running, Role: LEADER
I20260928 06:40:07.872099 20899 consensus_queue.cc:237] T 3d276b83b5df4001a4b6c875f2ca96a8 P 91c3a3280e414ab4a758e8cdedb881c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } }
I20260928 06:40:07.872992 20391 catalog_manager.cc:5495] T 3d276b83b5df4001a4b6c875f2ca96a8 P 91c3a3280e414ab4a758e8cdedb881c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91c3a3280e414ab4a758e8cdedb881c9 (127.18.211.67). New cstate: current_term: 1 leader_uuid: "91c3a3280e414ab4a758e8cdedb881c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } health_report { overall_health: HEALTHY } } }
I20260928 06:40:07.882453 20891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:07.880289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46678 (local address 127.18.211.67:40927)
0928 06:40:07.880338 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:07.880339 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:07.880347 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:40:07.880416 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:07.880417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:07.880432 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:07.880484 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:40:07.880598 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.881040 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.881781 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.881979 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.882027 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:07.882180 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:40:07.882268 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:40:07.882324 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:07.882357 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 06:40:07.882373 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:40:07.884398 20688 raft_consensus.cc:1261] T 3d276b83b5df4001a4b6c875f2ca96a8 P b1e75abdf38c4d00a00b7d8a0f506eb3 [term 1 FOLLOWER]: Refusing update from remote peer 91c3a3280e414ab4a758e8cdedb881c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:40:07.884398 20548 raft_consensus.cc:1261] T 3d276b83b5df4001a4b6c875f2ca96a8 P 2d71c58ff2f040cdbde917b51cb265a5 [term 1 FOLLOWER]: Refusing update from remote peer 91c3a3280e414ab4a758e8cdedb881c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:40:07.884707 20904 consensus_queue.cc:1035] T 3d276b83b5df4001a4b6c875f2ca96a8 P 91c3a3280e414ab4a758e8cdedb881c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:40:07.884855 20899 consensus_queue.cc:1035] T 3d276b83b5df4001a4b6c875f2ca96a8 P 91c3a3280e414ab4a758e8cdedb881c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:40:07.887132 20907 mvcc.cc:205] Tried to move back new op lower bound from 7334205881892237312 to 7334205881844682752. Current Snapshot: MvccSnapshot[applied={T|T < 7334205881892237312}]
I20260928 06:40:07.887312 20908 mvcc.cc:205] Tried to move back new op lower bound from 7334205881892237312 to 7334205881844682752. Current Snapshot: MvccSnapshot[applied={T|T < 7334205881892237312}]
I20260928 06:40:07.887624 20909 mvcc.cc:205] Tried to move back new op lower bound from 7334205881892237312 to 7334205881844682752. Current Snapshot: MvccSnapshot[applied={T|T < 7334205881892237312}]
I20260928 06:40:07.891516 20390 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:40:07.891625 20390 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:40:07.893278 20390 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38fcbd2db9a0413da62e6eadc0f9b220: Sending DeleteTablet for 3 replicas of tablet 3d276b83b5df4001a4b6c875f2ca96a8
I20260928 06:40:07.893700 20808 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d276b83b5df4001a4b6c875f2ca96a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:07 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:46662
I20260928 06:40:07.893745 20528 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d276b83b5df4001a4b6c875f2ca96a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:07 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:49572
I20260928 06:40:07.893932 20917 tablet_replica.cc:330] T 3d276b83b5df4001a4b6c875f2ca96a8 P 91c3a3280e414ab4a758e8cdedb881c9: stopping tablet replica
I20260928 06:40:07.893977 20668 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d276b83b5df4001a4b6c875f2ca96a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:07 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:56828
I20260928 06:40:07.894022 20917 raft_consensus.cc:2229] T 3d276b83b5df4001a4b6c875f2ca96a8 P 91c3a3280e414ab4a758e8cdedb881c9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:40:07.894153 20917 raft_consensus.cc:2258] T 3d276b83b5df4001a4b6c875f2ca96a8 P 91c3a3280e414ab4a758e8cdedb881c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:07.894170 20919 tablet_replica.cc:330] T 3d276b83b5df4001a4b6c875f2ca96a8 P b1e75abdf38c4d00a00b7d8a0f506eb3: stopping tablet replica
I20260928 06:40:07.894661 20917 ts_tablet_manager.cc:1905] T 3d276b83b5df4001a4b6c875f2ca96a8 P 91c3a3280e414ab4a758e8cdedb881c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:07.894263 20919 raft_consensus.cc:2229] T 3d276b83b5df4001a4b6c875f2ca96a8 P b1e75abdf38c4d00a00b7d8a0f506eb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:40:07.895175 20919 raft_consensus.cc:2258] T 3d276b83b5df4001a4b6c875f2ca96a8 P b1e75abdf38c4d00a00b7d8a0f506eb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:07.895628 20919 ts_tablet_manager.cc:1905] T 3d276b83b5df4001a4b6c875f2ca96a8 P b1e75abdf38c4d00a00b7d8a0f506eb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:07.896428 20917 ts_tablet_manager.cc:1918] T 3d276b83b5df4001a4b6c875f2ca96a8 P 91c3a3280e414ab4a758e8cdedb881c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:40:07.896494 20917 log.cc:1196] T 3d276b83b5df4001a4b6c875f2ca96a8 P 91c3a3280e414ab4a758e8cdedb881c9: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-2/wal/wals/3d276b83b5df4001a4b6c875f2ca96a8
I20260928 06:40:07.896746 20917 ts_tablet_manager.cc:1939] T 3d276b83b5df4001a4b6c875f2ca96a8 P 91c3a3280e414ab4a758e8cdedb881c9: Deleting consensus metadata
I20260928 06:40:07.897241 20378 catalog_manager.cc:4841] TS 91c3a3280e414ab4a758e8cdedb881c9 (127.18.211.67:40927): tablet 3d276b83b5df4001a4b6c875f2ca96a8 (table test-table [id=cab1989f0bff4928b4ed1c108cab8b4e]) successfully deleted
I20260928 06:40:07.897591 20919 ts_tablet_manager.cc:1918] T 3d276b83b5df4001a4b6c875f2ca96a8 P b1e75abdf38c4d00a00b7d8a0f506eb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:40:07.897655 20919 log.cc:1196] T 3d276b83b5df4001a4b6c875f2ca96a8 P b1e75abdf38c4d00a00b7d8a0f506eb3: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-1/wal/wals/3d276b83b5df4001a4b6c875f2ca96a8
I20260928 06:40:07.897737 20390 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 06:40:07.897876 20919 ts_tablet_manager.cc:1939] T 3d276b83b5df4001a4b6c875f2ca96a8 P b1e75abdf38c4d00a00b7d8a0f506eb3: Deleting consensus metadata
W20260928 06:40:07.897902 20390 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:40:07.898504 20918 tablet_replica.cc:330] T 3d276b83b5df4001a4b6c875f2ca96a8 P 2d71c58ff2f040cdbde917b51cb265a5: stopping tablet replica
I20260928 06:40:07.898584 20375 catalog_manager.cc:4841] TS b1e75abdf38c4d00a00b7d8a0f506eb3 (127.18.211.66:34597): tablet 3d276b83b5df4001a4b6c875f2ca96a8 (table test-table [id=cab1989f0bff4928b4ed1c108cab8b4e]) successfully deleted
I20260928 06:40:07.898969 20918 raft_consensus.cc:2229] T 3d276b83b5df4001a4b6c875f2ca96a8 P 2d71c58ff2f040cdbde917b51cb265a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:40:07.899433 20918 raft_consensus.cc:2258] T 3d276b83b5df4001a4b6c875f2ca96a8 P 2d71c58ff2f040cdbde917b51cb265a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:07.899762 20808 tablet_service.cc:1467] Processing CreateTablet for tablet 2a3b0127e0244fa88c610ec6e4928a6d (DEFAULT_TABLE table=test-table [id=8599b62065aa452db9e35edaae1be59f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:07.899876 20808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a3b0127e0244fa88c610ec6e4928a6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:07.899884 20918 ts_tablet_manager.cc:1905] T 3d276b83b5df4001a4b6c875f2ca96a8 P 2d71c58ff2f040cdbde917b51cb265a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:07.899906 20528 tablet_service.cc:1467] Processing CreateTablet for tablet 2a3b0127e0244fa88c610ec6e4928a6d (DEFAULT_TABLE table=test-table [id=8599b62065aa452db9e35edaae1be59f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:07.899991 20528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a3b0127e0244fa88c610ec6e4928a6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:07.901057 20894 tablet_bootstrap.cc:492] T 2a3b0127e0244fa88c610ec6e4928a6d P 91c3a3280e414ab4a758e8cdedb881c9: Bootstrap starting.
I20260928 06:40:07.901201 20918 ts_tablet_manager.cc:1918] T 3d276b83b5df4001a4b6c875f2ca96a8 P 2d71c58ff2f040cdbde917b51cb265a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:40:07.901257 20918 log.cc:1196] T 3d276b83b5df4001a4b6c875f2ca96a8 P 2d71c58ff2f040cdbde917b51cb265a5: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-0/wal/wals/3d276b83b5df4001a4b6c875f2ca96a8
I20260928 06:40:07.901257 20668 tablet_service.cc:1467] Processing CreateTablet for tablet 2a3b0127e0244fa88c610ec6e4928a6d (DEFAULT_TABLE table=test-table [id=8599b62065aa452db9e35edaae1be59f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:07.901352 20668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a3b0127e0244fa88c610ec6e4928a6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:07.901434 20894 tablet_bootstrap.cc:654] T 2a3b0127e0244fa88c610ec6e4928a6d P 91c3a3280e414ab4a758e8cdedb881c9: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:07.901485 20918 ts_tablet_manager.cc:1939] T 3d276b83b5df4001a4b6c875f2ca96a8 P 2d71c58ff2f040cdbde917b51cb265a5: Deleting consensus metadata
I20260928 06:40:07.901746 20895 tablet_bootstrap.cc:492] T 2a3b0127e0244fa88c610ec6e4928a6d P 2d71c58ff2f040cdbde917b51cb265a5: Bootstrap starting.
I20260928 06:40:07.902113 20895 tablet_bootstrap.cc:654] T 2a3b0127e0244fa88c610ec6e4928a6d P 2d71c58ff2f040cdbde917b51cb265a5: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:07.902272 20898 tablet_bootstrap.cc:492] T 2a3b0127e0244fa88c610ec6e4928a6d P b1e75abdf38c4d00a00b7d8a0f506eb3: Bootstrap starting.
I20260928 06:40:07.902635 20898 tablet_bootstrap.cc:654] T 2a3b0127e0244fa88c610ec6e4928a6d P b1e75abdf38c4d00a00b7d8a0f506eb3: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:07.902639 20375 catalog_manager.cc:4841] TS 2d71c58ff2f040cdbde917b51cb265a5 (127.18.211.65:35019): tablet 3d276b83b5df4001a4b6c875f2ca96a8 (table test-table [id=cab1989f0bff4928b4ed1c108cab8b4e]) successfully deleted
I20260928 06:40:07.902871 20894 tablet_bootstrap.cc:492] T 2a3b0127e0244fa88c610ec6e4928a6d P 91c3a3280e414ab4a758e8cdedb881c9: No bootstrap required, opened a new log
I20260928 06:40:07.902920 20894 ts_tablet_manager.cc:1397] T 2a3b0127e0244fa88c610ec6e4928a6d P 91c3a3280e414ab4a758e8cdedb881c9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:40:07.903074 20894 raft_consensus.cc:348] T 2a3b0127e0244fa88c610ec6e4928a6d P 91c3a3280e414ab4a758e8cdedb881c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } }
I20260928 06:40:07.903175 20894 raft_consensus.cc:374] T 2a3b0127e0244fa88c610ec6e4928a6d P 91c3a3280e414ab4a758e8cdedb881c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:07.903204 20894 raft_consensus.cc:729] T 2a3b0127e0244fa88c610ec6e4928a6d P 91c3a3280e414ab4a758e8cdedb881c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91c3a3280e414ab4a758e8cdedb881c9, State: Initialized, Role: FOLLOWER
I20260928 06:40:07.903242 20894 consensus_queue.cc:260] T 2a3b0127e0244fa88c610ec6e4928a6d P 91c3a3280e414ab4a758e8cdedb881c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } }
I20260928 06:40:07.903257 20898 tablet_bootstrap.cc:492] T 2a3b0127e0244fa88c610ec6e4928a6d P b1e75abdf38c4d00a00b7d8a0f506eb3: No bootstrap required, opened a new log
I20260928 06:40:07.903299 20898 ts_tablet_manager.cc:1397] T 2a3b0127e0244fa88c610ec6e4928a6d P b1e75abdf38c4d00a00b7d8a0f506eb3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 06:40:07.903326 20894 ts_tablet_manager.cc:1428] T 2a3b0127e0244fa88c610ec6e4928a6d P 91c3a3280e414ab4a758e8cdedb881c9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:07.903440 20898 raft_consensus.cc:348] T 2a3b0127e0244fa88c610ec6e4928a6d P b1e75abdf38c4d00a00b7d8a0f506eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } }
I20260928 06:40:07.903496 20898 raft_consensus.cc:374] T 2a3b0127e0244fa88c610ec6e4928a6d P b1e75abdf38c4d00a00b7d8a0f506eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:07.903513 20898 raft_consensus.cc:729] T 2a3b0127e0244fa88c610ec6e4928a6d P b1e75abdf38c4d00a00b7d8a0f506eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1e75abdf38c4d00a00b7d8a0f506eb3, State: Initialized, Role: FOLLOWER
I20260928 06:40:07.903550 20895 tablet_bootstrap.cc:492] T 2a3b0127e0244fa88c610ec6e4928a6d P 2d71c58ff2f040cdbde917b51cb265a5: No bootstrap required, opened a new log
I20260928 06:40:07.903551 20898 consensus_queue.cc:260] T 2a3b0127e0244fa88c610ec6e4928a6d P b1e75abdf38c4d00a00b7d8a0f506eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } }
I20260928 06:40:07.903599 20895 ts_tablet_manager.cc:1397] T 2a3b0127e0244fa88c610ec6e4928a6d P 2d71c58ff2f040cdbde917b51cb265a5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:40:07.903640 20898 ts_tablet_manager.cc:1428] T 2a3b0127e0244fa88c610ec6e4928a6d P b1e75abdf38c4d00a00b7d8a0f506eb3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:07.903743 20895 raft_consensus.cc:348] T 2a3b0127e0244fa88c610ec6e4928a6d P 2d71c58ff2f040cdbde917b51cb265a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } }
I20260928 06:40:07.903805 20895 raft_consensus.cc:374] T 2a3b0127e0244fa88c610ec6e4928a6d P 2d71c58ff2f040cdbde917b51cb265a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:07.903823 20895 raft_consensus.cc:729] T 2a3b0127e0244fa88c610ec6e4928a6d P 2d71c58ff2f040cdbde917b51cb265a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d71c58ff2f040cdbde917b51cb265a5, State: Initialized, Role: FOLLOWER
I20260928 06:40:07.903862 20895 consensus_queue.cc:260] T 2a3b0127e0244fa88c610ec6e4928a6d P 2d71c58ff2f040cdbde917b51cb265a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } }
I20260928 06:40:07.903944 20895 ts_tablet_manager.cc:1428] T 2a3b0127e0244fa88c610ec6e4928a6d P 2d71c58ff2f040cdbde917b51cb265a5: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:40:07.942273 20899 raft_consensus.cc:482] T 2a3b0127e0244fa88c610ec6e4928a6d P 91c3a3280e414ab4a758e8cdedb881c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:40:07.942399 20899 raft_consensus.cc:504] T 2a3b0127e0244fa88c610ec6e4928a6d P 91c3a3280e414ab4a758e8cdedb881c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } }
I20260928 06:40:07.942548 20899 leader_election.cc:288] T 2a3b0127e0244fa88c610ec6e4928a6d P 91c3a3280e414ab4a758e8cdedb881c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d71c58ff2f040cdbde917b51cb265a5 (127.18.211.65:35019), b1e75abdf38c4d00a00b7d8a0f506eb3 (127.18.211.66:34597)
I20260928 06:40:07.942768 20688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a3b0127e0244fa88c610ec6e4928a6d" candidate_uuid: "91c3a3280e414ab4a758e8cdedb881c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" is_pre_election: true
I20260928 06:40:07.942754 20548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a3b0127e0244fa88c610ec6e4928a6d" candidate_uuid: "91c3a3280e414ab4a758e8cdedb881c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d71c58ff2f040cdbde917b51cb265a5" is_pre_election: true
I20260928 06:40:07.942862 20548 raft_consensus.cc:2454] T 2a3b0127e0244fa88c610ec6e4928a6d P 2d71c58ff2f040cdbde917b51cb265a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91c3a3280e414ab4a758e8cdedb881c9 in term 0.
I20260928 06:40:07.942862 20688 raft_consensus.cc:2454] T 2a3b0127e0244fa88c610ec6e4928a6d P b1e75abdf38c4d00a00b7d8a0f506eb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91c3a3280e414ab4a758e8cdedb881c9 in term 0.
I20260928 06:40:07.943018 20759 leader_election.cc:302] T 2a3b0127e0244fa88c610ec6e4928a6d P 91c3a3280e414ab4a758e8cdedb881c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d71c58ff2f040cdbde917b51cb265a5, 91c3a3280e414ab4a758e8cdedb881c9; no voters:
I20260928 06:40:07.943109 20899 raft_consensus.cc:2781] T 2a3b0127e0244fa88c610ec6e4928a6d P 91c3a3280e414ab4a758e8cdedb881c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:40:07.943148 20899 raft_consensus.cc:482] T 2a3b0127e0244fa88c610ec6e4928a6d P 91c3a3280e414ab4a758e8cdedb881c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:40:07.943168 20899 raft_consensus.cc:3037] T 2a3b0127e0244fa88c610ec6e4928a6d P 91c3a3280e414ab4a758e8cdedb881c9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:07.943770 20899 raft_consensus.cc:504] T 2a3b0127e0244fa88c610ec6e4928a6d P 91c3a3280e414ab4a758e8cdedb881c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } }
I20260928 06:40:07.943879 20899 leader_election.cc:288] T 2a3b0127e0244fa88c610ec6e4928a6d P 91c3a3280e414ab4a758e8cdedb881c9 [CANDIDATE]: Term 1 election: Requested vote from peers 2d71c58ff2f040cdbde917b51cb265a5 (127.18.211.65:35019), b1e75abdf38c4d00a00b7d8a0f506eb3 (127.18.211.66:34597)
I20260928 06:40:07.944042 20548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a3b0127e0244fa88c610ec6e4928a6d" candidate_uuid: "91c3a3280e414ab4a758e8cdedb881c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d71c58ff2f040cdbde917b51cb265a5"
I20260928 06:40:07.944118 20548 raft_consensus.cc:3037] T 2a3b0127e0244fa88c610ec6e4928a6d P 2d71c58ff2f040cdbde917b51cb265a5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:07.944082 20688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a3b0127e0244fa88c610ec6e4928a6d" candidate_uuid: "91c3a3280e414ab4a758e8cdedb881c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3"
I20260928 06:40:07.944154 20688 raft_consensus.cc:3037] T 2a3b0127e0244fa88c610ec6e4928a6d P b1e75abdf38c4d00a00b7d8a0f506eb3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:07.944692 20548 raft_consensus.cc:2454] T 2a3b0127e0244fa88c610ec6e4928a6d P 2d71c58ff2f040cdbde917b51cb265a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91c3a3280e414ab4a758e8cdedb881c9 in term 1.
I20260928 06:40:07.944892 20688 raft_consensus.cc:2454] T 2a3b0127e0244fa88c610ec6e4928a6d P b1e75abdf38c4d00a00b7d8a0f506eb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91c3a3280e414ab4a758e8cdedb881c9 in term 1.
I20260928 06:40:07.944892 20759 leader_election.cc:302] T 2a3b0127e0244fa88c610ec6e4928a6d P 91c3a3280e414ab4a758e8cdedb881c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d71c58ff2f040cdbde917b51cb265a5, 91c3a3280e414ab4a758e8cdedb881c9; no voters:
I20260928 06:40:07.944977 20899 raft_consensus.cc:2781] T 2a3b0127e0244fa88c610ec6e4928a6d P 91c3a3280e414ab4a758e8cdedb881c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:07.945019 20899 raft_consensus.cc:686] T 2a3b0127e0244fa88c610ec6e4928a6d P 91c3a3280e414ab4a758e8cdedb881c9 [term 1 LEADER]: Becoming Leader. State: Replica: 91c3a3280e414ab4a758e8cdedb881c9, State: Running, Role: LEADER
I20260928 06:40:07.945060 20899 consensus_queue.cc:237] T 2a3b0127e0244fa88c610ec6e4928a6d P 91c3a3280e414ab4a758e8cdedb881c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } }
I20260928 06:40:07.945633 20390 catalog_manager.cc:5495] T 2a3b0127e0244fa88c610ec6e4928a6d P 91c3a3280e414ab4a758e8cdedb881c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91c3a3280e414ab4a758e8cdedb881c9 (127.18.211.67). New cstate: current_term: 1 leader_uuid: "91c3a3280e414ab4a758e8cdedb881c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } health_report { overall_health: HEALTHY } } }
Sep 28 06:40:07 dist-test-slave-b28r krb5kdc[20346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577607, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:07.974857 20929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:07.967869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35495 (local address 127.0.0.1:36694)
0928 06:40:07.968049 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:07.968058 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:07.968125 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:07.968246 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:07.968247 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:07.968591 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:07.968645 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:07.968648 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.969262 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.969263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:07.969775 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:07.969776 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.970087 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.970088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:07.970147 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:07.970770 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:07.970775 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:07.972375 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:07.973836 (+ 1461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:07.973843 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:07.973846 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:07.974024 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:07.974186 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:07.974186 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:07.974187 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:07.974241 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:07.974485 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:07.974493 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:07.974538 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 06:40:07.974601 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 06:40:07.974730 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":2,"thread_start_us":120,"threads_started":1}
I20260928 06:40:07.974862 20440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:07.967919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36694 (local address 127.18.211.126:35495)
0928 06:40:07.967965 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:07.967967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:07.968118 (+ 151us) server_negotiation.cc:407] Connection header received
0928 06:40:07.968181 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:07.968182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:07.968192 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:07.968242 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:07.968721 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.969194 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.969847 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:07.970017 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:07.970070 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:07.972457 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:07.972458 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:07.972459 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:07.972466 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:07.973745 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:07.974087 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:07.974088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:07.974088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:07.974135 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:07.974327 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:07.974327 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:07.974328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:07.974429 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:07.974484 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:07.974630 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 06:40:07.974789 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":23}
I20260928 06:40:07.977051 20390 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:36694:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 06:40:07.977274 20390 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:40:07.978570 20668 tablet_service.cc:1467] Processing CreateTablet for tablet 939c5eb933f84fa097f3654449ed733a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=93936994999b4350885eacf84a374cef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:40:07.978569 20528 tablet_service.cc:1467] Processing CreateTablet for tablet 939c5eb933f84fa097f3654449ed733a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=93936994999b4350885eacf84a374cef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:40:07.978580 20808 tablet_service.cc:1467] Processing CreateTablet for tablet 939c5eb933f84fa097f3654449ed733a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=93936994999b4350885eacf84a374cef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:40:07.978726 20528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 939c5eb933f84fa097f3654449ed733a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:07.978726 20668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 939c5eb933f84fa097f3654449ed733a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:07.978736 20808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 939c5eb933f84fa097f3654449ed733a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:07.980185 20898 tablet_bootstrap.cc:492] T 939c5eb933f84fa097f3654449ed733a P b1e75abdf38c4d00a00b7d8a0f506eb3: Bootstrap starting.
I20260928 06:40:07.980185 20895 tablet_bootstrap.cc:492] T 939c5eb933f84fa097f3654449ed733a P 2d71c58ff2f040cdbde917b51cb265a5: Bootstrap starting.
I20260928 06:40:07.980247 20894 tablet_bootstrap.cc:492] T 939c5eb933f84fa097f3654449ed733a P 91c3a3280e414ab4a758e8cdedb881c9: Bootstrap starting.
I20260928 06:40:07.980604 20895 tablet_bootstrap.cc:654] T 939c5eb933f84fa097f3654449ed733a P 2d71c58ff2f040cdbde917b51cb265a5: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:07.980604 20898 tablet_bootstrap.cc:654] T 939c5eb933f84fa097f3654449ed733a P b1e75abdf38c4d00a00b7d8a0f506eb3: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:07.981096 20894 tablet_bootstrap.cc:654] T 939c5eb933f84fa097f3654449ed733a P 91c3a3280e414ab4a758e8cdedb881c9: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:07.981336 20895 tablet_bootstrap.cc:492] T 939c5eb933f84fa097f3654449ed733a P 2d71c58ff2f040cdbde917b51cb265a5: No bootstrap required, opened a new log
I20260928 06:40:07.981384 20895 ts_tablet_manager.cc:1397] T 939c5eb933f84fa097f3654449ed733a P 2d71c58ff2f040cdbde917b51cb265a5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:07.981441 20898 tablet_bootstrap.cc:492] T 939c5eb933f84fa097f3654449ed733a P b1e75abdf38c4d00a00b7d8a0f506eb3: No bootstrap required, opened a new log
I20260928 06:40:07.981477 20898 ts_tablet_manager.cc:1397] T 939c5eb933f84fa097f3654449ed733a P b1e75abdf38c4d00a00b7d8a0f506eb3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 06:40:07.981524 20895 raft_consensus.cc:348] T 939c5eb933f84fa097f3654449ed733a P 2d71c58ff2f040cdbde917b51cb265a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } }
I20260928 06:40:07.981586 20895 raft_consensus.cc:374] T 939c5eb933f84fa097f3654449ed733a P 2d71c58ff2f040cdbde917b51cb265a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:07.981604 20895 raft_consensus.cc:729] T 939c5eb933f84fa097f3654449ed733a P 2d71c58ff2f040cdbde917b51cb265a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d71c58ff2f040cdbde917b51cb265a5, State: Initialized, Role: FOLLOWER
I20260928 06:40:07.981644 20895 consensus_queue.cc:260] T 939c5eb933f84fa097f3654449ed733a P 2d71c58ff2f040cdbde917b51cb265a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } }
I20260928 06:40:07.981639 20898 raft_consensus.cc:348] T 939c5eb933f84fa097f3654449ed733a P b1e75abdf38c4d00a00b7d8a0f506eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } }
I20260928 06:40:07.981694 20898 raft_consensus.cc:374] T 939c5eb933f84fa097f3654449ed733a P b1e75abdf38c4d00a00b7d8a0f506eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:07.981712 20898 raft_consensus.cc:729] T 939c5eb933f84fa097f3654449ed733a P b1e75abdf38c4d00a00b7d8a0f506eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1e75abdf38c4d00a00b7d8a0f506eb3, State: Initialized, Role: FOLLOWER
I20260928 06:40:07.981727 20895 ts_tablet_manager.cc:1428] T 939c5eb933f84fa097f3654449ed733a P 2d71c58ff2f040cdbde917b51cb265a5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:07.981747 20898 consensus_queue.cc:260] T 939c5eb933f84fa097f3654449ed733a P b1e75abdf38c4d00a00b7d8a0f506eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } }
I20260928 06:40:07.981825 20898 ts_tablet_manager.cc:1428] T 939c5eb933f84fa097f3654449ed733a P b1e75abdf38c4d00a00b7d8a0f506eb3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:07.981870 20894 tablet_bootstrap.cc:492] T 939c5eb933f84fa097f3654449ed733a P 91c3a3280e414ab4a758e8cdedb881c9: No bootstrap required, opened a new log
I20260928 06:40:07.981911 20894 ts_tablet_manager.cc:1397] T 939c5eb933f84fa097f3654449ed733a P 91c3a3280e414ab4a758e8cdedb881c9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 06:40:07.981889 20901 tablet_replica.cc:439] T 939c5eb933f84fa097f3654449ed733a P 2d71c58ff2f040cdbde917b51cb265a5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } } }
I20260928 06:40:07.981952 20901 tablet_replica.cc:442] T 939c5eb933f84fa097f3654449ed733a P 2d71c58ff2f040cdbde917b51cb265a5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:40:07.981981 20902 tablet_replica.cc:439] T 939c5eb933f84fa097f3654449ed733a P b1e75abdf38c4d00a00b7d8a0f506eb3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } } }
I20260928 06:40:07.982048 20902 tablet_replica.cc:442] T 939c5eb933f84fa097f3654449ed733a P b1e75abdf38c4d00a00b7d8a0f506eb3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:40:07.982038 20894 raft_consensus.cc:348] T 939c5eb933f84fa097f3654449ed733a P 91c3a3280e414ab4a758e8cdedb881c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } }
I20260928 06:40:07.982095 20894 raft_consensus.cc:374] T 939c5eb933f84fa097f3654449ed733a P 91c3a3280e414ab4a758e8cdedb881c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:07.982113 20894 raft_consensus.cc:729] T 939c5eb933f84fa097f3654449ed733a P 91c3a3280e414ab4a758e8cdedb881c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91c3a3280e414ab4a758e8cdedb881c9, State: Initialized, Role: FOLLOWER
I20260928 06:40:07.982157 20894 consensus_queue.cc:260] T 939c5eb933f84fa097f3654449ed733a P 91c3a3280e414ab4a758e8cdedb881c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } }
I20260928 06:40:07.982232 20894 ts_tablet_manager.cc:1428] T 939c5eb933f84fa097f3654449ed733a P 91c3a3280e414ab4a758e8cdedb881c9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:07.982502 20912 tablet_replica.cc:439] T 939c5eb933f84fa097f3654449ed733a P 91c3a3280e414ab4a758e8cdedb881c9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } } }
I20260928 06:40:07.982568 20912 tablet_replica.cc:442] T 939c5eb933f84fa097f3654449ed733a P 91c3a3280e414ab4a758e8cdedb881c9: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 06:40:08.008627 20595 tablet.cc:2369] T 2a3b0127e0244fa88c610ec6e4928a6d P 2d71c58ff2f040cdbde917b51cb265a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:40:08.018256 20912 raft_consensus.cc:482] T 939c5eb933f84fa097f3654449ed733a P 91c3a3280e414ab4a758e8cdedb881c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:40:08.018347 20912 raft_consensus.cc:504] T 939c5eb933f84fa097f3654449ed733a P 91c3a3280e414ab4a758e8cdedb881c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } }
I20260928 06:40:08.018487 20912 leader_election.cc:288] T 939c5eb933f84fa097f3654449ed733a P 91c3a3280e414ab4a758e8cdedb881c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d71c58ff2f040cdbde917b51cb265a5 (127.18.211.65:35019), b1e75abdf38c4d00a00b7d8a0f506eb3 (127.18.211.66:34597)
I20260928 06:40:08.018680 20548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "939c5eb933f84fa097f3654449ed733a" candidate_uuid: "91c3a3280e414ab4a758e8cdedb881c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d71c58ff2f040cdbde917b51cb265a5" is_pre_election: true
I20260928 06:40:08.018715 20688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "939c5eb933f84fa097f3654449ed733a" candidate_uuid: "91c3a3280e414ab4a758e8cdedb881c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" is_pre_election: true
I20260928 06:40:08.018785 20548 raft_consensus.cc:2454] T 939c5eb933f84fa097f3654449ed733a P 2d71c58ff2f040cdbde917b51cb265a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91c3a3280e414ab4a758e8cdedb881c9 in term 0.
I20260928 06:40:08.018812 20688 raft_consensus.cc:2454] T 939c5eb933f84fa097f3654449ed733a P b1e75abdf38c4d00a00b7d8a0f506eb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91c3a3280e414ab4a758e8cdedb881c9 in term 0.
I20260928 06:40:08.018940 20759 leader_election.cc:302] T 939c5eb933f84fa097f3654449ed733a P 91c3a3280e414ab4a758e8cdedb881c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d71c58ff2f040cdbde917b51cb265a5, 91c3a3280e414ab4a758e8cdedb881c9; no voters:
I20260928 06:40:08.019034 20912 raft_consensus.cc:2781] T 939c5eb933f84fa097f3654449ed733a P 91c3a3280e414ab4a758e8cdedb881c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:40:08.019095 20912 raft_consensus.cc:482] T 939c5eb933f84fa097f3654449ed733a P 91c3a3280e414ab4a758e8cdedb881c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:40:08.019116 20912 raft_consensus.cc:3037] T 939c5eb933f84fa097f3654449ed733a P 91c3a3280e414ab4a758e8cdedb881c9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:08.019738 20912 raft_consensus.cc:504] T 939c5eb933f84fa097f3654449ed733a P 91c3a3280e414ab4a758e8cdedb881c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } }
I20260928 06:40:08.019840 20912 leader_election.cc:288] T 939c5eb933f84fa097f3654449ed733a P 91c3a3280e414ab4a758e8cdedb881c9 [CANDIDATE]: Term 1 election: Requested vote from peers 2d71c58ff2f040cdbde917b51cb265a5 (127.18.211.65:35019), b1e75abdf38c4d00a00b7d8a0f506eb3 (127.18.211.66:34597)
I20260928 06:40:08.020028 20548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "939c5eb933f84fa097f3654449ed733a" candidate_uuid: "91c3a3280e414ab4a758e8cdedb881c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d71c58ff2f040cdbde917b51cb265a5"
I20260928 06:40:08.020047 20688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "939c5eb933f84fa097f3654449ed733a" candidate_uuid: "91c3a3280e414ab4a758e8cdedb881c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3"
I20260928 06:40:08.020115 20688 raft_consensus.cc:3037] T 939c5eb933f84fa097f3654449ed733a P b1e75abdf38c4d00a00b7d8a0f506eb3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:08.020118 20548 raft_consensus.cc:3037] T 939c5eb933f84fa097f3654449ed733a P 2d71c58ff2f040cdbde917b51cb265a5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:08.020684 20688 raft_consensus.cc:2454] T 939c5eb933f84fa097f3654449ed733a P b1e75abdf38c4d00a00b7d8a0f506eb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91c3a3280e414ab4a758e8cdedb881c9 in term 1.
I20260928 06:40:08.020777 20548 raft_consensus.cc:2454] T 939c5eb933f84fa097f3654449ed733a P 2d71c58ff2f040cdbde917b51cb265a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91c3a3280e414ab4a758e8cdedb881c9 in term 1.
I20260928 06:40:08.020844 20759 leader_election.cc:302] T 939c5eb933f84fa097f3654449ed733a P 91c3a3280e414ab4a758e8cdedb881c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91c3a3280e414ab4a758e8cdedb881c9, b1e75abdf38c4d00a00b7d8a0f506eb3; no voters:
I20260928 06:40:08.020927 20912 raft_consensus.cc:2781] T 939c5eb933f84fa097f3654449ed733a P 91c3a3280e414ab4a758e8cdedb881c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:08.020980 20912 raft_consensus.cc:686] T 939c5eb933f84fa097f3654449ed733a P 91c3a3280e414ab4a758e8cdedb881c9 [term 1 LEADER]: Becoming Leader. State: Replica: 91c3a3280e414ab4a758e8cdedb881c9, State: Running, Role: LEADER
I20260928 06:40:08.021026 20912 consensus_queue.cc:237] T 939c5eb933f84fa097f3654449ed733a P 91c3a3280e414ab4a758e8cdedb881c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } }
I20260928 06:40:08.021327 20899 tablet_replica.cc:439] T 939c5eb933f84fa097f3654449ed733a P 91c3a3280e414ab4a758e8cdedb881c9: TxnStatusTablet state changed. Reason: New leader 91c3a3280e414ab4a758e8cdedb881c9. Latest consensus state: current_term: 1 leader_uuid: "91c3a3280e414ab4a758e8cdedb881c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } } }
I20260928 06:40:08.021400 20899 tablet_replica.cc:442] T 939c5eb933f84fa097f3654449ed733a P 91c3a3280e414ab4a758e8cdedb881c9: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:40:08.021569 20934 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 06:40:08.021692 20934 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 06:40:08.021636 20391 catalog_manager.cc:5495] T 939c5eb933f84fa097f3654449ed733a P 91c3a3280e414ab4a758e8cdedb881c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 91c3a3280e414ab4a758e8cdedb881c9 (127.18.211.67). New cstate: current_term: 1 leader_uuid: "91c3a3280e414ab4a758e8cdedb881c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } health_report { overall_health: HEALTHY } } }
I20260928 06:40:08.031726 20891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:08.029316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46690 (local address 127.18.211.67:40927)
0928 06:40:08.029363 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:08.029365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:08.029410 (+ 45us) server_negotiation.cc:407] Connection header received
0928 06:40:08.029470 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:08.029471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:08.029483 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:08.029533 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:40:08.029651 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:08.030237 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:08.031017 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:08.031268 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:08.031334 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:08.031499 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:40:08.031533 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:40:08.031574 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:08.031630 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 06:40:08.031653 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:40:08.031741 20929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:08.029283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:40927 (local address 127.0.0.1:46690)
0928 06:40:08.029339 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:08.029352 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:08.029411 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:08.029541 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:08.029542 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:08.029543 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:40:08.029592 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:08.029594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:08.030324 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:08.030325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:08.030943 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:08.030945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:08.031340 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:08.031341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:08.031408 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:08.031424 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:40:08.031579 (+ 155us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:40:08.031587 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 06:40:08.031627 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 06:40:08.031674 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260928 06:40:08.032610 20688 raft_consensus.cc:1261] T 939c5eb933f84fa097f3654449ed733a P b1e75abdf38c4d00a00b7d8a0f506eb3 [term 1 FOLLOWER]: Refusing update from remote peer 91c3a3280e414ab4a758e8cdedb881c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:40:08.032651 20548 raft_consensus.cc:1261] T 939c5eb933f84fa097f3654449ed733a P 2d71c58ff2f040cdbde917b51cb265a5 [term 1 FOLLOWER]: Refusing update from remote peer 91c3a3280e414ab4a758e8cdedb881c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:40:08.032857 20912 consensus_queue.cc:1035] T 939c5eb933f84fa097f3654449ed733a P 91c3a3280e414ab4a758e8cdedb881c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 06:40:08.032945 20875 tablet.cc:2369] T 2a3b0127e0244fa88c610ec6e4928a6d P 91c3a3280e414ab4a758e8cdedb881c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:40:08.032984 20899 consensus_queue.cc:1035] T 939c5eb933f84fa097f3654449ed733a P 91c3a3280e414ab4a758e8cdedb881c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:40:08.033521 20901 tablet_replica.cc:439] T 939c5eb933f84fa097f3654449ed733a P 2d71c58ff2f040cdbde917b51cb265a5: TxnStatusTablet state changed. Reason: New leader 91c3a3280e414ab4a758e8cdedb881c9. Latest consensus state: current_term: 1 leader_uuid: "91c3a3280e414ab4a758e8cdedb881c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } } }
I20260928 06:40:08.033612 20901 tablet_replica.cc:442] T 939c5eb933f84fa097f3654449ed733a P 2d71c58ff2f040cdbde917b51cb265a5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:40:08.033550 20902 tablet_replica.cc:439] T 939c5eb933f84fa097f3654449ed733a P b1e75abdf38c4d00a00b7d8a0f506eb3: TxnStatusTablet state changed. Reason: New leader 91c3a3280e414ab4a758e8cdedb881c9. Latest consensus state: current_term: 1 leader_uuid: "91c3a3280e414ab4a758e8cdedb881c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } } }
I20260928 06:40:08.033641 20902 tablet_replica.cc:442] T 939c5eb933f84fa097f3654449ed733a P b1e75abdf38c4d00a00b7d8a0f506eb3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:40:08.034224 20912 tablet_replica.cc:439] T 939c5eb933f84fa097f3654449ed733a P 91c3a3280e414ab4a758e8cdedb881c9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "91c3a3280e414ab4a758e8cdedb881c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } } }
I20260928 06:40:08.034302 20912 tablet_replica.cc:442] T 939c5eb933f84fa097f3654449ed733a P 91c3a3280e414ab4a758e8cdedb881c9: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:40:08.034392 20901 tablet_replica.cc:439] T 939c5eb933f84fa097f3654449ed733a P 2d71c58ff2f040cdbde917b51cb265a5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "91c3a3280e414ab4a758e8cdedb881c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } } }
I20260928 06:40:08.034471 20901 tablet_replica.cc:442] T 939c5eb933f84fa097f3654449ed733a P 2d71c58ff2f040cdbde917b51cb265a5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:40:08.035808 20904 tablet_replica.cc:439] T 939c5eb933f84fa097f3654449ed733a P 91c3a3280e414ab4a758e8cdedb881c9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "91c3a3280e414ab4a758e8cdedb881c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } } }
I20260928 06:40:08.035892 20904 tablet_replica.cc:442] T 939c5eb933f84fa097f3654449ed733a P 91c3a3280e414ab4a758e8cdedb881c9: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:40:08.036339 20902 tablet_replica.cc:439] T 939c5eb933f84fa097f3654449ed733a P b1e75abdf38c4d00a00b7d8a0f506eb3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "91c3a3280e414ab4a758e8cdedb881c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 } } peers { permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 } } peers { permanent_uuid: "91c3a3280e414ab4a758e8cdedb881c9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40927 } } }
I20260928 06:40:08.036424 20902 tablet_replica.cc:442] T 939c5eb933f84fa097f3654449ed733a P b1e75abdf38c4d00a00b7d8a0f506eb3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:40:08.040164 20891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:08.038087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:55401 (local address 127.18.211.67:40927)
0928 06:40:08.038122 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:08.038124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:08.038133 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:40:08.038167 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:08.038168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:08.038176 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:08.038212 (+ 36us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:40:08.038339 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:08.038823 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:08.039574 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:08.039756 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:08.039820 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:08.039953 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:40:08.039984 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:40:08.040022 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:08.040075 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 06:40:08.040090 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:40:08.040169 20880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:08.038012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:40927 (local address 127.18.211.67:55401)
0928 06:40:08.038071 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:08.038082 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:08.038121 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:08.038234 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:08.038235 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:08.038237 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:40:08.038282 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:08.038283 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:08.038886 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:08.038887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:08.039503 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:08.039505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:08.039823 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:08.039824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:08.039873 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:08.039887 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:40:08.040028 (+ 141us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:40:08.040029 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 06:40:08.040065 (+ 36us) client_negotiation.cc:240] Negotiation successful
0928 06:40:08.040105 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260928 06:40:08.042088 20688 raft_consensus.cc:1261] T 2a3b0127e0244fa88c610ec6e4928a6d P b1e75abdf38c4d00a00b7d8a0f506eb3 [term 1 FOLLOWER]: Refusing update from remote peer 91c3a3280e414ab4a758e8cdedb881c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:40:08.042093 20548 raft_consensus.cc:1261] T 2a3b0127e0244fa88c610ec6e4928a6d P 2d71c58ff2f040cdbde917b51cb265a5 [term 1 FOLLOWER]: Refusing update from remote peer 91c3a3280e414ab4a758e8cdedb881c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:40:08.042287 20904 consensus_queue.cc:1035] T 2a3b0127e0244fa88c610ec6e4928a6d P 91c3a3280e414ab4a758e8cdedb881c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d71c58ff2f040cdbde917b51cb265a5" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:40:08.042395 20912 consensus_queue.cc:1035] T 2a3b0127e0244fa88c610ec6e4928a6d P 91c3a3280e414ab4a758e8cdedb881c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1e75abdf38c4d00a00b7d8a0f506eb3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 34597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:40:08.056200 20391 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:40:08.056311 20391 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:40:08.056886 20391 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38fcbd2db9a0413da62e6eadc0f9b220: Sending DeleteTablet for 3 replicas of tablet 2a3b0127e0244fa88c610ec6e4928a6d
I20260928 06:40:08.057119 20808 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a3b0127e0244fa88c610ec6e4928a6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:08 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:46662
I20260928 06:40:08.057220 20917 tablet_replica.cc:330] T 2a3b0127e0244fa88c610ec6e4928a6d P 91c3a3280e414ab4a758e8cdedb881c9: stopping tablet replica
I20260928 06:40:08.057201 20528 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a3b0127e0244fa88c610ec6e4928a6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:08 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:49572
I20260928 06:40:08.057265 20917 raft_consensus.cc:2229] T 2a3b0127e0244fa88c610ec6e4928a6d P 91c3a3280e414ab4a758e8cdedb881c9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:40:08.057268 20668 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a3b0127e0244fa88c610ec6e4928a6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:08 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:56828
I20260928 06:40:08.057310 20917 raft_consensus.cc:2258] T 2a3b0127e0244fa88c610ec6e4928a6d P 91c3a3280e414ab4a758e8cdedb881c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:08.057380 20919 tablet_replica.cc:330] T 2a3b0127e0244fa88c610ec6e4928a6d P b1e75abdf38c4d00a00b7d8a0f506eb3: stopping tablet replica
I20260928 06:40:08.057403 20918 tablet_replica.cc:330] T 2a3b0127e0244fa88c610ec6e4928a6d P 2d71c58ff2f040cdbde917b51cb265a5: stopping tablet replica
I20260928 06:40:08.057432 20919 raft_consensus.cc:2229] T 2a3b0127e0244fa88c610ec6e4928a6d P b1e75abdf38c4d00a00b7d8a0f506eb3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:40:08.057448 20918 raft_consensus.cc:2229] T 2a3b0127e0244fa88c610ec6e4928a6d P 2d71c58ff2f040cdbde917b51cb265a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:40:08.057464 20919 raft_consensus.cc:2258] T 2a3b0127e0244fa88c610ec6e4928a6d P b1e75abdf38c4d00a00b7d8a0f506eb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:08.057478 20918 raft_consensus.cc:2258] T 2a3b0127e0244fa88c610ec6e4928a6d P 2d71c58ff2f040cdbde917b51cb265a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:08.057765 20919 ts_tablet_manager.cc:1905] T 2a3b0127e0244fa88c610ec6e4928a6d P b1e75abdf38c4d00a00b7d8a0f506eb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:08.057780 20918 ts_tablet_manager.cc:1905] T 2a3b0127e0244fa88c610ec6e4928a6d P 2d71c58ff2f040cdbde917b51cb265a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:08.057906 20917 ts_tablet_manager.cc:1905] T 2a3b0127e0244fa88c610ec6e4928a6d P 91c3a3280e414ab4a758e8cdedb881c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:08.059733 20918 ts_tablet_manager.cc:1918] T 2a3b0127e0244fa88c610ec6e4928a6d P 2d71c58ff2f040cdbde917b51cb265a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:40:08.059772 20917 ts_tablet_manager.cc:1918] T 2a3b0127e0244fa88c610ec6e4928a6d P 91c3a3280e414ab4a758e8cdedb881c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:40:08.059793 20918 log.cc:1196] T 2a3b0127e0244fa88c610ec6e4928a6d P 2d71c58ff2f040cdbde917b51cb265a5: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-0/wal/wals/2a3b0127e0244fa88c610ec6e4928a6d
I20260928 06:40:08.059823 20917 log.cc:1196] T 2a3b0127e0244fa88c610ec6e4928a6d P 91c3a3280e414ab4a758e8cdedb881c9: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-2/wal/wals/2a3b0127e0244fa88c610ec6e4928a6d
I20260928 06:40:08.059998 20918 ts_tablet_manager.cc:1939] T 2a3b0127e0244fa88c610ec6e4928a6d P 2d71c58ff2f040cdbde917b51cb265a5: Deleting consensus metadata
I20260928 06:40:08.060019 20917 ts_tablet_manager.cc:1939] T 2a3b0127e0244fa88c610ec6e4928a6d P 91c3a3280e414ab4a758e8cdedb881c9: Deleting consensus metadata
I20260928 06:40:08.060277 20919 ts_tablet_manager.cc:1918] T 2a3b0127e0244fa88c610ec6e4928a6d P b1e75abdf38c4d00a00b7d8a0f506eb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:40:08.060288 20375 catalog_manager.cc:4841] TS 2d71c58ff2f040cdbde917b51cb265a5 (127.18.211.65:35019): tablet 2a3b0127e0244fa88c610ec6e4928a6d (table test-table [id=8599b62065aa452db9e35edaae1be59f]) successfully deleted
I20260928 06:40:08.060321 20378 catalog_manager.cc:4841] TS 91c3a3280e414ab4a758e8cdedb881c9 (127.18.211.67:40927): tablet 2a3b0127e0244fa88c610ec6e4928a6d (table test-table [id=8599b62065aa452db9e35edaae1be59f]) successfully deleted
I20260928 06:40:08.060343 20919 log.cc:1196] T 2a3b0127e0244fa88c610ec6e4928a6d P b1e75abdf38c4d00a00b7d8a0f506eb3: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790577602522428-19277-0/minicluster-data/ts-1/wal/wals/2a3b0127e0244fa88c610ec6e4928a6d
I20260928 06:40:08.060518 20919 ts_tablet_manager.cc:1939] T 2a3b0127e0244fa88c610ec6e4928a6d P b1e75abdf38c4d00a00b7d8a0f506eb3: Deleting consensus metadata
I20260928 06:40:08.060789 20375 catalog_manager.cc:4841] TS b1e75abdf38c4d00a00b7d8a0f506eb3 (127.18.211.66:34597): tablet 2a3b0127e0244fa88c610ec6e4928a6d (table test-table [id=8599b62065aa452db9e35edaae1be59f]) successfully deleted
Sep 28 06:40:08 dist-test-slave-b28r krb5kdc[20346](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577607, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.211.65@KRBTEST.COM
I20260928 06:40:08.069351 20890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:08.060900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49582 (local address 127.18.211.65:35019)
0928 06:40:08.060941 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:08.060943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:08.061057 (+ 114us) server_negotiation.cc:407] Connection header received
0928 06:40:08.061096 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:08.061097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:08.061106 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:08.061168 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:08.061637 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:08.062371 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:08.063073 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:08.063411 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:08.063478 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:08.066303 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:08.066312 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:08.066313 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:08.066320 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:08.068157 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:08.068487 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:08.068491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:08.068492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:08.068561 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:08.068769 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:08.068769 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:08.068770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:08.068945 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:08.069097 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:08.069175 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 06:40:08.069254 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 06:40:08.069561 20588 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49582
I20260928 06:40:08.075838 20440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:08.070478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36702 (local address 127.18.211.126:35495)
0928 06:40:08.070525 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:08.070527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:08.070535 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:40:08.070592 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:08.070593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:08.070603 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:08.070654 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:08.071138 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:08.071653 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:08.072301 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:08.072501 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:08.072554 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:08.073563 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:08.073564 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:08.073564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:08.073572 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:08.074847 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:08.075168 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:08.075169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:08.075170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:08.075226 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:08.075386 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:08.075387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:08.075387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:08.075496 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:08.075577 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:08.075674 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 06:40:08.075749 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260928 06:40:08.076043 20391 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36702
I20260928 06:40:08.076573 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 20463
I20260928 06:40:08.081605 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 20603
I20260928 06:40:08.086169 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 20743
I20260928 06:40:08.091109 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 20359
2026-09-28T06:40:08Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (1962 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:40:08 dist-test-slave-b28r krb5kdc[20950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:40:08 dist-test-slave-b28r krb5kdc[20950](info): set up 2 sockets
Sep 28 06:40:08 dist-test-slave-b28r krb5kdc[20950](info): commencing operation
krb5kdc: starting...
W20260928 06:40:09.043577 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:40:09 dist-test-slave-b28r krb5kdc[20950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:40:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:40:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.211.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
I20260928 06:40:09.152060 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:37569
--webserver_interface=127.18.211.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:33441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:37569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:09.229825 20963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:09.230023 20963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:09.230052 20963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:09.231238 20963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:40:09.231278 20963 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 06:40:09.231290 20963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:09.231302 20963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:09.231312 20963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:40:09.231323 20963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:40:09.232777 20963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:33441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:37569
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:37569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:09.232995 20963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:09.233209 20963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:09.238022 20971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:09.238015 20963 server_base.cc:1034] running on GCE node
W20260928 06:40:09.237986 20969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:09.237980 20968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:09.238446 20963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:09.238729 20963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:09.239876 20963 hybrid_clock.cc:648] HybridClock initialized: now 1790577609239860 us; error 35 us; skew 500 ppm
Sep 28 06:40:09 dist-test-slave-b28r krb5kdc[20950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577609, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:09.245296 20963 init.cc:377] Logged in from keytab as kudu/127.18.211.126@KRBTEST.COM (short username kudu)
I20260928 06:40:09.245813 20963 webserver.cc:460] Webserver started at http://127.18.211.126:45881/ using document root <none> and password file <none>
I20260928 06:40:09.246037 20963 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:09.246088 20963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:09.246213 20963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:09.247321 20963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "3b309eba9ace4bb4a324ccd7217e4227"
format_stamp: "Formatted at 2026-09-28 06:40:09 on dist-test-slave-b28r"
server_key: "ffb923def971241a096465f96cd060ab"
server_key_iv: "b8678609ca8656bc51f83eeaf1b0821b"
server_key_version: "encryptionkey@0"
I20260928 06:40:09.247663 20963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "3b309eba9ace4bb4a324ccd7217e4227"
format_stamp: "Formatted at 2026-09-28 06:40:09 on dist-test-slave-b28r"
server_key: "ffb923def971241a096465f96cd060ab"
server_key_iv: "b8678609ca8656bc51f83eeaf1b0821b"
server_key_version: "encryptionkey@0"
I20260928 06:40:09.248941 20963 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:40:09.249691 20978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:09.249959 20963 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:09.250053 20963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "3b309eba9ace4bb4a324ccd7217e4227"
format_stamp: "Formatted at 2026-09-28 06:40:09 on dist-test-slave-b28r"
server_key: "ffb923def971241a096465f96cd060ab"
server_key_iv: "b8678609ca8656bc51f83eeaf1b0821b"
server_key_version: "encryptionkey@0"
I20260928 06:40:09.250118 20963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:09.274256 20963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:09.276232 20963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:09.276376 20963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:09.281011 20963 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:37569
I20260928 06:40:09.281077 21040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:37569 every 8 connection(s)
I20260928 06:40:09.281385 20963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:40:09.282133 21041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:09.284431 21041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b309eba9ace4bb4a324ccd7217e4227: Bootstrap starting.
I20260928 06:40:09.284991 21041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b309eba9ace4bb4a324ccd7217e4227: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:09.285257 21041 log.cc:824] T 00000000000000000000000000000000 P 3b309eba9ace4bb4a324ccd7217e4227: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:09.285954 21041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b309eba9ace4bb4a324ccd7217e4227: No bootstrap required, opened a new log
I20260928 06:40:09.286849 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 20963
I20260928 06:40:09.286944 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:40:09.287137 19277 external_mini_cluster.cc:1428] Setting key d59309f4d35b0e30234e4fd346fa4a81
I20260928 06:40:09.287921 21041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b309eba9ace4bb4a324ccd7217e4227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b309eba9ace4bb4a324ccd7217e4227" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 37569 } }
I20260928 06:40:09.288061 21041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b309eba9ace4bb4a324ccd7217e4227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:09.288082 21041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b309eba9ace4bb4a324ccd7217e4227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b309eba9ace4bb4a324ccd7217e4227, State: Initialized, Role: FOLLOWER
I20260928 06:40:09.288198 21041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b309eba9ace4bb4a324ccd7217e4227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b309eba9ace4bb4a324ccd7217e4227" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 37569 } }
I20260928 06:40:09.288268 21041 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b309eba9ace4bb4a324ccd7217e4227 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:40:09.288295 21041 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b309eba9ace4bb4a324ccd7217e4227 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:40:09.288336 21041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b309eba9ace4bb4a324ccd7217e4227 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:09.288931 21041 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b309eba9ace4bb4a324ccd7217e4227 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b309eba9ace4bb4a324ccd7217e4227" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 37569 } }
I20260928 06:40:09.289031 21041 leader_election.cc:302] T 00000000000000000000000000000000 P 3b309eba9ace4bb4a324ccd7217e4227 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b309eba9ace4bb4a324ccd7217e4227; no voters:
I20260928 06:40:09.289177 21041 leader_election.cc:288] T 00000000000000000000000000000000 P 3b309eba9ace4bb4a324ccd7217e4227 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:40:09.289213 21046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b309eba9ace4bb4a324ccd7217e4227 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:09.289386 21046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b309eba9ace4bb4a324ccd7217e4227 [term 1 LEADER]: Becoming Leader. State: Replica: 3b309eba9ace4bb4a324ccd7217e4227, State: Running, Role: LEADER
I20260928 06:40:09.289425 21041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b309eba9ace4bb4a324ccd7217e4227 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:40:09.289474 21046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b309eba9ace4bb4a324ccd7217e4227 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b309eba9ace4bb4a324ccd7217e4227" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 37569 } }
I20260928 06:40:09.289848 21046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b309eba9ace4bb4a324ccd7217e4227 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b309eba9ace4bb4a324ccd7217e4227. Latest consensus state: current_term: 1 leader_uuid: "3b309eba9ace4bb4a324ccd7217e4227" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b309eba9ace4bb4a324ccd7217e4227" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 37569 } } }
I20260928 06:40:09.289923 21046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b309eba9ace4bb4a324ccd7217e4227 [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:09.289917 21047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b309eba9ace4bb4a324ccd7217e4227 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b309eba9ace4bb4a324ccd7217e4227" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b309eba9ace4bb4a324ccd7217e4227" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 37569 } } }
I20260928 06:40:09.290007 21047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b309eba9ace4bb4a324ccd7217e4227 [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:09.290292 21050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:40:09.290815 21050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:40:09.292455 21050 catalog_manager.cc:1347] Generated new cluster ID: d6877b3be2f24c6da42c2f7850e03cb8
I20260928 06:40:09.292500 21050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:40:09 dist-test-slave-b28r krb5kdc[20950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:09.298504 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:09.287845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53610 (local address 127.18.211.126:37569)
0928 06:40:09.288019 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:09.288028 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:09.288055 (+ 27us) server_negotiation.cc:407] Connection header received
0928 06:40:09.288184 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:09.288199 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:09.288304 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:09.288449 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:09.289042 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.290639 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.291650 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.292205 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.292324 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:09.295047 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:09.295051 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:09.295051 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:09.295064 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:09.296955 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:09.297314 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:09.297317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:09.297318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:09.297446 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:09.297620 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:09.297621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:09.297621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:09.297937 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:09.298097 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:09.298165 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 06:40:09.298302 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
I20260928 06:40:09.303566 21050 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:40:09.304037 21050 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:40:09.308408 21050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b309eba9ace4bb4a324ccd7217e4227: Generated new TSK 0
I20260928 06:40:09.308548 21050 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.211.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab.
Entry for principal kudu/127.18.211.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab.
I20260928 06:40:09.333791 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.65:0
--local_ip_for_outbound_sockets=127.18.211.65
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:37569
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:33441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:09.412417 21067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:09.412618 21067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:09.412643 21067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:09.413760 21067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:09.413800 21067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:09.413820 21067 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 06:40:09.413830 21067 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:40:09.413847 21067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.65
I20260928 06:40:09.415345 21067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:33441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.211.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.211.126:37569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.65
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:09.415552 21067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:09.415771 21067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:09.420519 21073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:09.420568 21067 server_base.cc:1034] running on GCE node
W20260928 06:40:09.420534 21072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:09.420516 21075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:09.420925 21067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:09.421119 21067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:09.422294 21067 hybrid_clock.cc:648] HybridClock initialized: now 1790577609422279 us; error 31 us; skew 500 ppm
Sep 28 06:40:09 dist-test-slave-b28r krb5kdc[20950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577609, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:09.424660 21067 init.cc:377] Logged in from keytab as kudu/127.18.211.65@KRBTEST.COM (short username kudu)
I20260928 06:40:09.425062 21067 webserver.cc:460] Webserver started at http://127.18.211.65:40897/ using document root <none> and password file <none>
I20260928 06:40:09.425235 21067 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:09.425271 21067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:09.425360 21067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:09.426347 21067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-0/data/instance:
uuid: "5a402cacef4e43ca90ed68ac5f334659"
format_stamp: "Formatted at 2026-09-28 06:40:09 on dist-test-slave-b28r"
server_key: "4753112627322b947c785f7f35a314a4"
server_key_iv: "045207cac6c7bd223bf9295448b7e6f3"
server_key_version: "encryptionkey@0"
I20260928 06:40:09.426643 21067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance:
uuid: "5a402cacef4e43ca90ed68ac5f334659"
format_stamp: "Formatted at 2026-09-28 06:40:09 on dist-test-slave-b28r"
server_key: "4753112627322b947c785f7f35a314a4"
server_key_iv: "045207cac6c7bd223bf9295448b7e6f3"
server_key_version: "encryptionkey@0"
I20260928 06:40:09.427878 21067 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:40:09.428539 21082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:09.428682 21067 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:40:09.428761 21067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-0/wal
uuid: "5a402cacef4e43ca90ed68ac5f334659"
format_stamp: "Formatted at 2026-09-28 06:40:09 on dist-test-slave-b28r"
server_key: "4753112627322b947c785f7f35a314a4"
server_key_iv: "045207cac6c7bd223bf9295448b7e6f3"
server_key_version: "encryptionkey@0"
I20260928 06:40:09.428819 21067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:09.447746 21067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:09.449836 21067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:09.449975 21067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:09.450472 21067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:09.450517 21067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:09.450546 21067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:09.450563 21067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:40:09 dist-test-slave-b28r krb5kdc[20950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577609, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:09.457172 21067 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.65:41567
I20260928 06:40:09.457376 21197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.65:41567 every 8 connection(s)
I20260928 06:40:09.457556 21067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
I20260928 06:40:09.458467 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 21067
I20260928 06:40:09.458573 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance
I20260928 06:40:09.458763 19277 external_mini_cluster.cc:1428] Setting key 6d793b0c0d1801be565275551f893e8e
I20260928 06:40:09.460690 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:09.450989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:42449 (local address 127.18.211.126:37569)
0928 06:40:09.451042 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:09.451045 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:09.451276 (+ 231us) server_negotiation.cc:407] Connection header received
0928 06:40:09.451807 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:09.451809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:09.451828 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:09.451890 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:09.452708 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.453520 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.454994 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.455331 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.455414 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:09.457697 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:09.457699 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:09.457699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:09.457709 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:09.459335 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:09.459793 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:09.459794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:09.459794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:09.459852 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:09.460115 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:09.460116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:09.460116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:09.460220 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:09.460265 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:09.460551 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 06:40:09.460623 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 06:40:09.460924 21091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:09.450931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:37569 (local address 127.18.211.65:42449)
0928 06:40:09.451143 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:09.451191 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:09.451702 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:09.451972 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:09.451977 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:09.452315 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:09.452588 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:09.452595 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.453616 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.453617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:09.454893 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:09.454897 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.455419 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.455420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:09.455526 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:09.456040 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:09.456051 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:09.457605 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:09.459417 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:09.459424 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:09.459426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:09.459717 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:09.459914 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:09.459915 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:09.459915 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:09.460055 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:09.460270 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:09.460275 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:09.460426 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 06:40:09.460541 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 06:40:09.460702 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260928 06:40:09.461414 21198 heartbeater.cc:347] Connected to a master server at 127.18.211.126:37569
I20260928 06:40:09.461499 21198 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:09.461699 21198 heartbeater.cc:511] Master 127.18.211.126:37569 requested a full tablet report, sending...
I20260928 06:40:09.462275 20994 ts_manager.cc:194] Registered new tserver with Master: 5a402cacef4e43ca90ed68ac5f334659 (127.18.211.65:41567)
I20260928 06:40:09.462981 20994 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.65@KRBTEST.COM'} at 127.18.211.65:42449
WARNING: no policy specified for kudu/127.18.211.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.66@KRBTEST.COM" created.
I20260928 06:40:09.472084 21205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:09.462496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:37569 (local address 127.18.211.65:53263)
0928 06:40:09.466531 (+ 4035us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:09.466543 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:09.466614 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:09.466748 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:09.466749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:09.466901 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:09.466953 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:09.466955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.467672 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.467673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:09.468192 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:09.468193 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.468583 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.468584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:09.468674 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:09.469094 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:09.469102 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:09.469382 (+ 280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:09.471018 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:09.471019 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:09.471020 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:09.471290 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:09.471515 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:09.471516 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:09.471516 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:09.471571 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:09.471822 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:09.471823 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:09.471867 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 06:40:09.471935 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 06:40:09.472015 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4004,"thread_start_us":71,"threads_started":1}
I20260928 06:40:09.472126 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:09.462560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:53263 (local address 127.18.211.126:37569)
0928 06:40:09.462599 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:09.462601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:09.466611 (+ 4010us) server_negotiation.cc:407] Connection header received
0928 06:40:09.466665 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:09.466666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:09.466676 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:09.466732 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:09.467016 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.467586 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.468278 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.468507 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.468581 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:09.469480 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:09.469481 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:09.469481 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:09.469489 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:09.470938 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:09.471371 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:09.471372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:09.471373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:09.471457 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:09.471621 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:09.471622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:09.471622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:09.471741 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:09.471802 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:09.471966 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 06:40:09.472043 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab.
Entry for principal kudu/127.18.211.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab.
I20260928 06:40:09.482867 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.66:0
--local_ip_for_outbound_sockets=127.18.211.66
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:37569
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:33441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:09.564073 21207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:09.564273 21207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:09.564306 21207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:09.565395 21207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:09.565434 21207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:09.565461 21207 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 06:40:09.565477 21207 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:40:09.565507 21207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.66
I20260928 06:40:09.566879 21207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:33441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.211.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.211.126:37569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.66
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:09.567122 21207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:09.567324 21207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:09.571954 21213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:09.571973 21212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:09.571988 21215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:09.572005 21207 server_base.cc:1034] running on GCE node
I20260928 06:40:09.572441 21207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:09.572703 21207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:09.573853 21207 hybrid_clock.cc:648] HybridClock initialized: now 1790577609573837 us; error 30 us; skew 500 ppm
Sep 28 06:40:09 dist-test-slave-b28r krb5kdc[20950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577609, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:09.576579 21207 init.cc:377] Logged in from keytab as kudu/127.18.211.66@KRBTEST.COM (short username kudu)
I20260928 06:40:09.577025 21207 webserver.cc:460] Webserver started at http://127.18.211.66:37033/ using document root <none> and password file <none>
I20260928 06:40:09.577219 21207 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:09.577265 21207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:09.577401 21207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:09.578433 21207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-1/data/instance:
uuid: "f030cb407ac5452d9a76231f8449a3a4"
format_stamp: "Formatted at 2026-09-28 06:40:09 on dist-test-slave-b28r"
server_key: "5bc4782eac2f4cc8841cec4638b77d62"
server_key_iv: "3f4a3ffdd58a40d355cf5cf78759b7e9"
server_key_version: "encryptionkey@0"
I20260928 06:40:09.578753 21207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance:
uuid: "f030cb407ac5452d9a76231f8449a3a4"
format_stamp: "Formatted at 2026-09-28 06:40:09 on dist-test-slave-b28r"
server_key: "5bc4782eac2f4cc8841cec4638b77d62"
server_key_iv: "3f4a3ffdd58a40d355cf5cf78759b7e9"
server_key_version: "encryptionkey@0"
I20260928 06:40:09.580051 21207 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:40:09.580852 21222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:09.581015 21207 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:40:09.581096 21207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-1/wal
uuid: "f030cb407ac5452d9a76231f8449a3a4"
format_stamp: "Formatted at 2026-09-28 06:40:09 on dist-test-slave-b28r"
server_key: "5bc4782eac2f4cc8841cec4638b77d62"
server_key_iv: "3f4a3ffdd58a40d355cf5cf78759b7e9"
server_key_version: "encryptionkey@0"
I20260928 06:40:09.581152 21207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:09.612421 21207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:09.614387 21207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:09.614526 21207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:09.615039 21207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:09.615118 21207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:09.615160 21207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:09.615180 21207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:40:09 dist-test-slave-b28r krb5kdc[20950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577609, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:09.621857 21207 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.66:46747
I20260928 06:40:09.622030 21337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.66:46747 every 8 connection(s)
I20260928 06:40:09.622233 21207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
I20260928 06:40:09.624958 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:09.615573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:40295 (local address 127.18.211.126:37569)
0928 06:40:09.615621 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:09.615623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:09.615866 (+ 243us) server_negotiation.cc:407] Connection header received
0928 06:40:09.616365 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:09.616366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:09.616378 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:09.616424 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:09.617238 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.617986 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.619515 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.619764 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.619826 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:09.622318 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:09.622319 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:09.622320 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:09.622327 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:09.623717 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:09.624077 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:09.624078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:09.624078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:09.624137 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:09.624361 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:09.624362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:09.624362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:09.624478 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:09.624533 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:09.624791 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 06:40:09.624869 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:40:09.625123 21231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:09.615576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:37569 (local address 127.18.211.66:40295)
0928 06:40:09.615745 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:09.615781 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:09.616270 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:09.616501 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:09.616505 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:09.616852 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:09.617155 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:09.617163 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.618091 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.618092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:09.619420 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:09.619423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.619850 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.619851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:09.620003 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:09.620498 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:09.620511 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:09.622231 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:09.623792 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:09.623799 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:09.623801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:09.624011 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:09.624191 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:09.624191 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:09.624192 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:09.624305 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:09.624530 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:09.624537 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:09.624669 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 06:40:09.624759 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 06:40:09.624914 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":86,"threads_started":1}
I20260928 06:40:09.625603 21338 heartbeater.cc:347] Connected to a master server at 127.18.211.126:37569
I20260928 06:40:09.625685 21338 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:09.625859 21338 heartbeater.cc:511] Master 127.18.211.126:37569 requested a full tablet report, sending...
I20260928 06:40:09.626292 20995 ts_manager.cc:194] Registered new tserver with Master: f030cb407ac5452d9a76231f8449a3a4 (127.18.211.66:46747)
I20260928 06:40:09.626766 20995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.66@KRBTEST.COM'} at 127.18.211.66:40295
I20260928 06:40:09.627612 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 21207
I20260928 06:40:09.627694 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance
I20260928 06:40:09.627861 19277 external_mini_cluster.cc:1428] Setting key 71ee5204860566e2ae36c66c129d5748
I20260928 06:40:09.632141 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:09.626629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:55837 (local address 127.18.211.126:37569)
0928 06:40:09.626669 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:09.626671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:09.626721 (+ 50us) server_negotiation.cc:407] Connection header received
0928 06:40:09.626783 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:09.626784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:09.626791 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:09.626835 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:09.627304 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.627920 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.628680 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.628864 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.628915 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:09.629696 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:09.629698 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:09.629698 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:09.629704 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:09.630977 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:09.631305 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:09.631307 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:09.631307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:09.631386 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:09.631593 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:09.631594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:09.631594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:09.631740 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:09.631825 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:09.631980 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 06:40:09.632049 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:40:09.632156 21344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:09.626552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:37569 (local address 127.18.211.66:55837)
0928 06:40:09.626662 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:09.626678 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:09.626734 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:09.626871 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:09.626872 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:09.627061 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:09.627168 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:09.627171 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.628001 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.628002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:09.628598 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:09.628600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.628919 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.628920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:09.628990 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:09.629362 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:09.629366 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:09.629620 (+ 254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:09.631049 (+ 1429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:09.631050 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:09.631050 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:09.631242 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:09.631453 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:09.631454 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:09.631454 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:09.631514 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:09.631841 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:09.631842 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:09.631889 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 06:40:09.631948 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 06:40:09.632050 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":25,"thread_start_us":61,"threads_started":1}
WARNING: no policy specified for kudu/127.18.211.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab.
Entry for principal kudu/127.18.211.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab.
I20260928 06:40:09.650739 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.67:0
--local_ip_for_outbound_sockets=127.18.211.67
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:37569
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:33441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:09.728360 21347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:09.728549 21347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:09.728579 21347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:09.729672 21347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:09.729717 21347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:09.729748 21347 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 06:40:09.729763 21347 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:40:09.729784 21347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.67
I20260928 06:40:09.731242 21347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:33441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.211.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.211.126:37569
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.67
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:09.731467 21347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:09.731663 21347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:09.736351 21355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:09.736449 21352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:09.736357 21353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:09.736575 21347 server_base.cc:1034] running on GCE node
I20260928 06:40:09.736745 21347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:09.736953 21347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:09.738085 21347 hybrid_clock.cc:648] HybridClock initialized: now 1790577609738069 us; error 28 us; skew 500 ppm
Sep 28 06:40:09 dist-test-slave-b28r krb5kdc[20950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577609, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:09.740484 21347 init.cc:377] Logged in from keytab as kudu/127.18.211.67@KRBTEST.COM (short username kudu)
I20260928 06:40:09.740849 21347 webserver.cc:460] Webserver started at http://127.18.211.67:44679/ using document root <none> and password file <none>
I20260928 06:40:09.741029 21347 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:09.741066 21347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:09.741153 21347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:09.742115 21347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-2/data/instance:
uuid: "20a3dd6652e74d1b8d6890637a957a51"
format_stamp: "Formatted at 2026-09-28 06:40:09 on dist-test-slave-b28r"
server_key: "500ab516231b0aa9aff7e605f9d674ed"
server_key_iv: "3fb91d3dcef4db9627133399e6b0f203"
server_key_version: "encryptionkey@0"
I20260928 06:40:09.742401 21347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance:
uuid: "20a3dd6652e74d1b8d6890637a957a51"
format_stamp: "Formatted at 2026-09-28 06:40:09 on dist-test-slave-b28r"
server_key: "500ab516231b0aa9aff7e605f9d674ed"
server_key_iv: "3fb91d3dcef4db9627133399e6b0f203"
server_key_version: "encryptionkey@0"
I20260928 06:40:09.743683 21347 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:40:09.744333 21362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:09.744482 21347 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:40:09.744558 21347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-2/wal
uuid: "20a3dd6652e74d1b8d6890637a957a51"
format_stamp: "Formatted at 2026-09-28 06:40:09 on dist-test-slave-b28r"
server_key: "500ab516231b0aa9aff7e605f9d674ed"
server_key_iv: "3fb91d3dcef4db9627133399e6b0f203"
server_key_version: "encryptionkey@0"
I20260928 06:40:09.744617 21347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:09.762470 21347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:09.764331 21347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:09.764436 21347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:09.764866 21347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:09.764904 21347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:09.764925 21347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:09.764938 21347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:09.771327 21347 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.67:34267
I20260928 06:40:09.771385 21477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.67:34267 every 8 connection(s)
Sep 28 06:40:09 dist-test-slave-b28r krb5kdc[20950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577609, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:09.771721 21347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
I20260928 06:40:09.774492 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:09.765336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:47663 (local address 127.18.211.126:37569)
0928 06:40:09.765385 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:09.765387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:09.765614 (+ 227us) server_negotiation.cc:407] Connection header received
0928 06:40:09.766136 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:09.766137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:09.766153 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:09.766262 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:09.767036 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.767793 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.769282 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.769586 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.769663 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:09.772168 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:09.772169 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:09.772169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:09.772178 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:09.773406 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:09.773746 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:09.773746 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:09.773747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:09.773791 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:09.773987 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:09.773988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:09.773989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:09.774107 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:09.774169 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:09.774369 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 06:40:09.774429 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:40:09.774622 21371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:09.765344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:37569 (local address 127.18.211.67:47663)
0928 06:40:09.765501 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:09.765536 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:09.766018 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:09.766341 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:09.766345 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:09.766664 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:09.766940 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:09.766946 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.767914 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.767915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:09.769159 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:09.769163 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.769667 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.769668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:09.769794 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:09.770281 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:09.770291 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:09.772062 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:09.773477 (+ 1415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:09.773483 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:09.773484 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:09.773686 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:09.773828 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:09.773829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:09.773829 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:09.773916 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:09.774163 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:09.774165 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:09.774271 (+ 106us) client_negotiation.cc:769] Sending connection context
0928 06:40:09.774347 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 06:40:09.774454 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260928 06:40:09.775070 21478 heartbeater.cc:347] Connected to a master server at 127.18.211.126:37569
I20260928 06:40:09.775175 21478 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:09.775249 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 21347
I20260928 06:40:09.775354 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance
I20260928 06:40:09.775408 21478 heartbeater.cc:511] Master 127.18.211.126:37569 requested a full tablet report, sending...
I20260928 06:40:09.775519 19277 external_mini_cluster.cc:1428] Setting key 7a209f3c0931208385ddcc2fd3fc5ec7
I20260928 06:40:09.775753 20994 ts_manager.cc:194] Registered new tserver with Master: 20a3dd6652e74d1b8d6890637a957a51 (127.18.211.67:34267)
I20260928 06:40:09.776265 20994 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.67@KRBTEST.COM'} at 127.18.211.67:47663
I20260928 06:40:09.776362 19277 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:40:09.780941 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:09.776210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:48045 (local address 127.18.211.126:37569)
0928 06:40:09.776252 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:09.776254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:09.776332 (+ 78us) server_negotiation.cc:407] Connection header received
0928 06:40:09.776358 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:09.776359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:09.776382 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:09.776454 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:09.776798 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.777268 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.777917 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.778111 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.778161 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:09.778912 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:09.778913 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:09.778914 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:09.778920 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:09.780004 (+ 1084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:09.780290 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:09.780291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:09.780292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:09.780345 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:09.780501 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:09.780501 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:09.780502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:09.780600 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:09.780653 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:09.780787 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 06:40:09.780852 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:40:09.780953 21484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:09.776146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:37569 (local address 127.18.211.67:48045)
0928 06:40:09.776256 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:09.776265 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:09.776320 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:09.776496 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:09.776496 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:09.776669 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:09.776728 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:09.776729 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.777340 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.777341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:09.777849 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:09.777850 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.778165 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.778166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:09.778235 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:09.778586 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:09.778589 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:09.778839 (+ 250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:09.780072 (+ 1233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:09.780073 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:09.780074 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:09.780233 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:09.780397 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:09.780397 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:09.780398 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:09.780451 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:09.780651 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:09.780651 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:09.780694 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 06:40:09.780759 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 06:40:09.780854 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":63,"threads_started":1}
Sep 28 06:40:09 dist-test-slave-b28r krb5kdc[20950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:40:09 dist-test-slave-b28r krb5kdc[20950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577609, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:09.797708 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:09.791432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53618 (local address 127.18.211.126:37569)
0928 06:40:09.791470 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:09.791471 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:09.791478 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:40:09.791498 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:09.791499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:09.791507 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:09.791552 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:09.792047 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.792532 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.793190 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.793384 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.793427 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:09.795503 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:09.795505 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:09.795505 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:09.795512 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:09.796713 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:09.797038 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:09.797039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:09.797039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:09.797100 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:09.797321 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:09.797322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:09.797322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:09.797433 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:09.797517 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:09.797589 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 06:40:09.797643 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:40:09.799383 20994 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53618:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:40:09.800901 20994 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:40:09.807354 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:09.803204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:41567 (local address 127.0.0.1:49684)
0928 06:40:09.803414 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:09.803423 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:09.803466 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:09.803695 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:09.803698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:09.803701 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:09.803766 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:09.803767 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.805096 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.805097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:09.806139 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:09.806142 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.806997 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.806999 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:09.807126 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:09.807139 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 06:40:09.807214 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 06:40:09.807269 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":3,"thread_start_us":49,"threads_started":1}
I20260928 06:40:09.807360 21492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:09.802979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:46747 (local address 127.0.0.1:54890)
0928 06:40:09.803129 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:09.803169 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:09.803238 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:09.803749 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:09.803750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:09.803752 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:09.803810 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:09.803812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.804792 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.804793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:09.806041 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:09.806044 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.806998 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.806999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:09.807122 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:09.807139 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 06:40:09.807268 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 06:40:09.807301 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":2,"thread_start_us":60,"threads_started":1}
I20260928 06:40:09.807386 21496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:09.803339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49684 (local address 127.18.211.65:41567)
0928 06:40:09.803503 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:09.803508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:09.803517 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:40:09.803554 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:09.803556 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:09.803572 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:09.803843 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:09.803905 (+ 62us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.805010 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.806204 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.806897 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.806973 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:09.807032 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:09.807251 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 06:40:09.807291 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
I20260928 06:40:09.807628 21494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:09.803124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54890 (local address 127.18.211.66:46747)
0928 06:40:09.803547 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:09.803552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:09.803564 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:40:09.803608 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:09.803610 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:09.803627 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:09.803894 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:09.803950 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.804712 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.806131 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.806898 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.806973 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:09.807032 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:09.807493 (+ 461us) server_negotiation.cc:299] Negotiation successful
0928 06:40:09.807538 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":134,"threads_started":1}
I20260928 06:40:09.808070 21132 tablet_service.cc:1467] Processing CreateTablet for tablet 9ca657f7b26d4bfba92974472491a794 (DEFAULT_TABLE table=test-table [id=eafea5d540ad46f6827a834a63e47376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:09.808436 21132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ca657f7b26d4bfba92974472491a794. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:09.808753 21495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:09.803330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:34267 (local address 127.0.0.1:42698)
0928 06:40:09.803629 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:09.803640 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:09.803697 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:09.804017 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:09.804017 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:09.804020 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:09.804054 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:09.804055 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.805216 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.805217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:09.806057 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:09.806060 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.808511 (+ 2451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.808512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:09.808569 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:09.808573 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:40:09.808625 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 06:40:09.808665 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":42,"threads_started":1}
I20260928 06:40:09.808773 21497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:09.803443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42698 (local address 127.18.211.67:34267)
0928 06:40:09.803861 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:09.803865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:09.803874 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:40:09.803908 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:09.803910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:09.803924 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:09.804000 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:09.804119 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.805148 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.807067 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.808439 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.808500 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:09.808555 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:09.808638 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 06:40:09.808685 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":84,"threads_started":1}
I20260928 06:40:09.809310 21412 tablet_service.cc:1467] Processing CreateTablet for tablet 9ca657f7b26d4bfba92974472491a794 (DEFAULT_TABLE table=test-table [id=eafea5d540ad46f6827a834a63e47376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:09.809594 21412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ca657f7b26d4bfba92974472491a794. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:09.811563 21498 tablet_bootstrap.cc:492] T 9ca657f7b26d4bfba92974472491a794 P 5a402cacef4e43ca90ed68ac5f334659: Bootstrap starting.
I20260928 06:40:09.811738 21272 tablet_service.cc:1467] Processing CreateTablet for tablet 9ca657f7b26d4bfba92974472491a794 (DEFAULT_TABLE table=test-table [id=eafea5d540ad46f6827a834a63e47376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:09.812080 21272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ca657f7b26d4bfba92974472491a794. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:09.812218 21498 tablet_bootstrap.cc:654] T 9ca657f7b26d4bfba92974472491a794 P 5a402cacef4e43ca90ed68ac5f334659: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:09.812427 21499 tablet_bootstrap.cc:492] T 9ca657f7b26d4bfba92974472491a794 P 20a3dd6652e74d1b8d6890637a957a51: Bootstrap starting.
I20260928 06:40:09.812543 21498 log.cc:824] T 9ca657f7b26d4bfba92974472491a794 P 5a402cacef4e43ca90ed68ac5f334659: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:09.813323 21498 tablet_bootstrap.cc:492] T 9ca657f7b26d4bfba92974472491a794 P 5a402cacef4e43ca90ed68ac5f334659: No bootstrap required, opened a new log
I20260928 06:40:09.813334 21499 tablet_bootstrap.cc:654] T 9ca657f7b26d4bfba92974472491a794 P 20a3dd6652e74d1b8d6890637a957a51: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:09.813396 21498 ts_tablet_manager.cc:1397] T 9ca657f7b26d4bfba92974472491a794 P 5a402cacef4e43ca90ed68ac5f334659: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:40:09.813660 21499 log.cc:824] T 9ca657f7b26d4bfba92974472491a794 P 20a3dd6652e74d1b8d6890637a957a51: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:09.814384 21499 tablet_bootstrap.cc:492] T 9ca657f7b26d4bfba92974472491a794 P 20a3dd6652e74d1b8d6890637a957a51: No bootstrap required, opened a new log
I20260928 06:40:09.814452 21499 ts_tablet_manager.cc:1397] T 9ca657f7b26d4bfba92974472491a794 P 20a3dd6652e74d1b8d6890637a957a51: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:40:09.815101 21502 tablet_bootstrap.cc:492] T 9ca657f7b26d4bfba92974472491a794 P f030cb407ac5452d9a76231f8449a3a4: Bootstrap starting.
I20260928 06:40:09.815701 21502 tablet_bootstrap.cc:654] T 9ca657f7b26d4bfba92974472491a794 P f030cb407ac5452d9a76231f8449a3a4: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:09.815750 21498 raft_consensus.cc:348] T 9ca657f7b26d4bfba92974472491a794 P 5a402cacef4e43ca90ed68ac5f334659 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:09.815919 21498 raft_consensus.cc:374] T 9ca657f7b26d4bfba92974472491a794 P 5a402cacef4e43ca90ed68ac5f334659 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:09.815959 21498 raft_consensus.cc:729] T 9ca657f7b26d4bfba92974472491a794 P 5a402cacef4e43ca90ed68ac5f334659 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a402cacef4e43ca90ed68ac5f334659, State: Initialized, Role: FOLLOWER
I20260928 06:40:09.815982 21502 log.cc:824] T 9ca657f7b26d4bfba92974472491a794 P f030cb407ac5452d9a76231f8449a3a4: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:09.816088 21498 consensus_queue.cc:260] T 9ca657f7b26d4bfba92974472491a794 P 5a402cacef4e43ca90ed68ac5f334659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:09.816372 21198 heartbeater.cc:503] Master 127.18.211.126:37569 was elected leader, sending a full tablet report...
I20260928 06:40:09.816387 21499 raft_consensus.cc:348] T 9ca657f7b26d4bfba92974472491a794 P 20a3dd6652e74d1b8d6890637a957a51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:09.816533 21499 raft_consensus.cc:374] T 9ca657f7b26d4bfba92974472491a794 P 20a3dd6652e74d1b8d6890637a957a51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:09.816572 21499 raft_consensus.cc:729] T 9ca657f7b26d4bfba92974472491a794 P 20a3dd6652e74d1b8d6890637a957a51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a3dd6652e74d1b8d6890637a957a51, State: Initialized, Role: FOLLOWER
I20260928 06:40:09.816694 21498 ts_tablet_manager.cc:1428] T 9ca657f7b26d4bfba92974472491a794 P 5a402cacef4e43ca90ed68ac5f334659: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:40:09.816704 21499 consensus_queue.cc:260] T 9ca657f7b26d4bfba92974472491a794 P 20a3dd6652e74d1b8d6890637a957a51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:09.816756 21502 tablet_bootstrap.cc:492] T 9ca657f7b26d4bfba92974472491a794 P f030cb407ac5452d9a76231f8449a3a4: No bootstrap required, opened a new log
I20260928 06:40:09.816811 21502 ts_tablet_manager.cc:1397] T 9ca657f7b26d4bfba92974472491a794 P f030cb407ac5452d9a76231f8449a3a4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:40:09.816955 21499 ts_tablet_manager.cc:1428] T 9ca657f7b26d4bfba92974472491a794 P 20a3dd6652e74d1b8d6890637a957a51: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:40:09.816977 21478 heartbeater.cc:503] Master 127.18.211.126:37569 was elected leader, sending a full tablet report...
I20260928 06:40:09.818569 21502 raft_consensus.cc:348] T 9ca657f7b26d4bfba92974472491a794 P f030cb407ac5452d9a76231f8449a3a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:09.818688 21502 raft_consensus.cc:374] T 9ca657f7b26d4bfba92974472491a794 P f030cb407ac5452d9a76231f8449a3a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:09.818717 21502 raft_consensus.cc:729] T 9ca657f7b26d4bfba92974472491a794 P f030cb407ac5452d9a76231f8449a3a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f030cb407ac5452d9a76231f8449a3a4, State: Initialized, Role: FOLLOWER
I20260928 06:40:09.818830 21502 consensus_queue.cc:260] T 9ca657f7b26d4bfba92974472491a794 P f030cb407ac5452d9a76231f8449a3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:09.819063 21502 ts_tablet_manager.cc:1428] T 9ca657f7b26d4bfba92974472491a794 P f030cb407ac5452d9a76231f8449a3a4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:40:09.819157 21338 heartbeater.cc:503] Master 127.18.211.126:37569 was elected leader, sending a full tablet report...
W20260928 06:40:09.872812 21339 tablet.cc:2369] T 9ca657f7b26d4bfba92974472491a794 P f030cb407ac5452d9a76231f8449a3a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:40:09.881174 21506 raft_consensus.cc:482] T 9ca657f7b26d4bfba92974472491a794 P f030cb407ac5452d9a76231f8449a3a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:40:09.881320 21506 raft_consensus.cc:504] T 9ca657f7b26d4bfba92974472491a794 P f030cb407ac5452d9a76231f8449a3a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:09.881670 21506 leader_election.cc:288] T 9ca657f7b26d4bfba92974472491a794 P f030cb407ac5452d9a76231f8449a3a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a402cacef4e43ca90ed68ac5f334659 (127.18.211.65:41567), 20a3dd6652e74d1b8d6890637a957a51 (127.18.211.67:34267)
I20260928 06:40:09.884809 21497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:09.882040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:33561 (local address 127.18.211.67:34267)
0928 06:40:09.882092 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:09.882093 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:09.882228 (+ 135us) server_negotiation.cc:407] Connection header received
0928 06:40:09.882254 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:09.882255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:09.882265 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:09.882308 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:09.882449 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.882889 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.883850 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.884443 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.884512 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:09.884521 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:09.884709 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 06:40:09.884747 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:40:09.884860 21496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:09.881935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:38543 (local address 127.18.211.65:41567)
0928 06:40:09.881984 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:09.881986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:09.881997 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:40:09.882026 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:09.882026 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:09.882036 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:09.882098 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:09.882301 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.882854 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.883892 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.884442 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.884512 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:09.884521 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:09.884747 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 06:40:09.884795 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:40:09.884860 21231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:09.881858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:41567 (local address 127.18.211.66:38543)
0928 06:40:09.881909 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:09.881921 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:09.881963 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:09.882126 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:09.882127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:09.882129 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:09.882191 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:09.882193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.882935 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.882936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:09.883794 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:09.883796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.884537 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.884538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:09.884611 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:09.884615 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:40:09.884709 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 06:40:09.884751 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":45}
I20260928 06:40:09.884855 21507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:09.881956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:34267 (local address 127.18.211.66:33561)
0928 06:40:09.882140 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:09.882150 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:09.882198 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:09.882319 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:09.882319 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:09.882322 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:09.882387 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:09.882388 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.882960 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.882961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:09.883768 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:09.883770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.884569 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.884570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:09.884639 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:09.884642 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:40:09.884687 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 06:40:09.884735 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":113,"threads_started":1}
I20260928 06:40:09.885289 21432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ca657f7b26d4bfba92974472491a794" candidate_uuid: "f030cb407ac5452d9a76231f8449a3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a3dd6652e74d1b8d6890637a957a51" is_pre_election: true
I20260928 06:40:09.885299 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ca657f7b26d4bfba92974472491a794" candidate_uuid: "f030cb407ac5452d9a76231f8449a3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a402cacef4e43ca90ed68ac5f334659" is_pre_election: true
I20260928 06:40:09.885442 21432 raft_consensus.cc:2454] T 9ca657f7b26d4bfba92974472491a794 P 20a3dd6652e74d1b8d6890637a957a51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f030cb407ac5452d9a76231f8449a3a4 in term 0.
I20260928 06:40:09.885443 21152 raft_consensus.cc:2454] T 9ca657f7b26d4bfba92974472491a794 P 5a402cacef4e43ca90ed68ac5f334659 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f030cb407ac5452d9a76231f8449a3a4 in term 0.
I20260928 06:40:09.885648 21224 leader_election.cc:302] T 9ca657f7b26d4bfba92974472491a794 P f030cb407ac5452d9a76231f8449a3a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a3dd6652e74d1b8d6890637a957a51, f030cb407ac5452d9a76231f8449a3a4; no voters:
I20260928 06:40:09.885771 21506 raft_consensus.cc:2781] T 9ca657f7b26d4bfba92974472491a794 P f030cb407ac5452d9a76231f8449a3a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:40:09.885828 21506 raft_consensus.cc:482] T 9ca657f7b26d4bfba92974472491a794 P f030cb407ac5452d9a76231f8449a3a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:40:09.885850 21506 raft_consensus.cc:3037] T 9ca657f7b26d4bfba92974472491a794 P f030cb407ac5452d9a76231f8449a3a4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:09.886708 21506 raft_consensus.cc:504] T 9ca657f7b26d4bfba92974472491a794 P f030cb407ac5452d9a76231f8449a3a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:09.886811 21506 leader_election.cc:288] T 9ca657f7b26d4bfba92974472491a794 P f030cb407ac5452d9a76231f8449a3a4 [CANDIDATE]: Term 1 election: Requested vote from peers 5a402cacef4e43ca90ed68ac5f334659 (127.18.211.65:41567), 20a3dd6652e74d1b8d6890637a957a51 (127.18.211.67:34267)
I20260928 06:40:09.886967 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ca657f7b26d4bfba92974472491a794" candidate_uuid: "f030cb407ac5452d9a76231f8449a3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a402cacef4e43ca90ed68ac5f334659"
I20260928 06:40:09.887010 21432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ca657f7b26d4bfba92974472491a794" candidate_uuid: "f030cb407ac5452d9a76231f8449a3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a3dd6652e74d1b8d6890637a957a51"
I20260928 06:40:09.887050 21152 raft_consensus.cc:3037] T 9ca657f7b26d4bfba92974472491a794 P 5a402cacef4e43ca90ed68ac5f334659 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:09.887106 21432 raft_consensus.cc:3037] T 9ca657f7b26d4bfba92974472491a794 P 20a3dd6652e74d1b8d6890637a957a51 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:09.887759 21152 raft_consensus.cc:2454] T 9ca657f7b26d4bfba92974472491a794 P 5a402cacef4e43ca90ed68ac5f334659 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f030cb407ac5452d9a76231f8449a3a4 in term 1.
I20260928 06:40:09.887789 21432 raft_consensus.cc:2454] T 9ca657f7b26d4bfba92974472491a794 P 20a3dd6652e74d1b8d6890637a957a51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f030cb407ac5452d9a76231f8449a3a4 in term 1.
I20260928 06:40:09.887926 21224 leader_election.cc:302] T 9ca657f7b26d4bfba92974472491a794 P f030cb407ac5452d9a76231f8449a3a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a402cacef4e43ca90ed68ac5f334659, f030cb407ac5452d9a76231f8449a3a4; no voters:
I20260928 06:40:09.888020 21506 raft_consensus.cc:2781] T 9ca657f7b26d4bfba92974472491a794 P f030cb407ac5452d9a76231f8449a3a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:09.888155 21506 raft_consensus.cc:686] T 9ca657f7b26d4bfba92974472491a794 P f030cb407ac5452d9a76231f8449a3a4 [term 1 LEADER]: Becoming Leader. State: Replica: f030cb407ac5452d9a76231f8449a3a4, State: Running, Role: LEADER
I20260928 06:40:09.888252 21506 consensus_queue.cc:237] T 9ca657f7b26d4bfba92974472491a794 P f030cb407ac5452d9a76231f8449a3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:09.889056 20995 catalog_manager.cc:5495] T 9ca657f7b26d4bfba92974472491a794 P f030cb407ac5452d9a76231f8449a3a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f030cb407ac5452d9a76231f8449a3a4 (127.18.211.66). New cstate: current_term: 1 leader_uuid: "f030cb407ac5452d9a76231f8449a3a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } health_report { overall_health: UNKNOWN } } }
Sep 28 06:40:09 dist-test-slave-b28r krb5kdc[20950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577609, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:09.926086 21515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:09.919807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:37569 (local address 127.0.0.1:53630)
0928 06:40:09.919989 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:09.919998 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:09.920070 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:09.920194 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:09.920195 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:09.920515 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:09.920561 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:09.920564 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.921209 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.921210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:09.921765 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:09.921767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.922079 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.922079 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:09.922128 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:09.922615 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:09.922619 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:09.923922 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:09.925127 (+ 1205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:09.925134 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:09.925137 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:09.925323 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:09.925486 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:09.925487 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:09.925487 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:09.925539 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:09.925739 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:09.925746 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:09.925793 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 06:40:09.925853 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 06:40:09.925970 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":84,"threads_started":1}
I20260928 06:40:09.926131 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:09.919867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53630 (local address 127.18.211.126:37569)
0928 06:40:09.919930 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:09.919932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:09.920069 (+ 137us) server_negotiation.cc:407] Connection header received
0928 06:40:09.920135 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:09.920136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:09.920144 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:09.920184 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:09.920644 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.921135 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.921842 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.922023 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.922076 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:09.924003 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:09.924004 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:09.924005 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:09.924011 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:09.925058 (+ 1047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:09.925385 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:09.925386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:09.925386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:09.925435 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:09.925587 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:09.925588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:09.925588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:09.925687 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:09.925742 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:09.925888 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 06:40:09.926032 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":38}
I20260928 06:40:09.927901 20995 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:53630:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 06:40:09.928084 20995 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:40:09.929512 21132 tablet_service.cc:1467] Processing CreateTablet for tablet 7c26f1a96a3047aea5359bfd06dc311f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d0d8c893090c4d2e9f7dedd12dd231ac]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:40:09.929543 21272 tablet_service.cc:1467] Processing CreateTablet for tablet 7c26f1a96a3047aea5359bfd06dc311f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d0d8c893090c4d2e9f7dedd12dd231ac]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:40:09.929592 21412 tablet_service.cc:1467] Processing CreateTablet for tablet 7c26f1a96a3047aea5359bfd06dc311f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d0d8c893090c4d2e9f7dedd12dd231ac]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:40:09.929702 21272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c26f1a96a3047aea5359bfd06dc311f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:09.929702 21132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c26f1a96a3047aea5359bfd06dc311f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:09.929709 21412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c26f1a96a3047aea5359bfd06dc311f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:09.930790 21502 tablet_bootstrap.cc:492] T 7c26f1a96a3047aea5359bfd06dc311f P f030cb407ac5452d9a76231f8449a3a4: Bootstrap starting.
I20260928 06:40:09.931145 21498 tablet_bootstrap.cc:492] T 7c26f1a96a3047aea5359bfd06dc311f P 5a402cacef4e43ca90ed68ac5f334659: Bootstrap starting.
I20260928 06:40:09.931176 21499 tablet_bootstrap.cc:492] T 7c26f1a96a3047aea5359bfd06dc311f P 20a3dd6652e74d1b8d6890637a957a51: Bootstrap starting.
I20260928 06:40:09.931227 21502 tablet_bootstrap.cc:654] T 7c26f1a96a3047aea5359bfd06dc311f P f030cb407ac5452d9a76231f8449a3a4: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:09.931706 21499 tablet_bootstrap.cc:654] T 7c26f1a96a3047aea5359bfd06dc311f P 20a3dd6652e74d1b8d6890637a957a51: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:09.931758 21498 tablet_bootstrap.cc:654] T 7c26f1a96a3047aea5359bfd06dc311f P 5a402cacef4e43ca90ed68ac5f334659: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:09.931953 21502 tablet_bootstrap.cc:492] T 7c26f1a96a3047aea5359bfd06dc311f P f030cb407ac5452d9a76231f8449a3a4: No bootstrap required, opened a new log
I20260928 06:40:09.932010 21502 ts_tablet_manager.cc:1397] T 7c26f1a96a3047aea5359bfd06dc311f P f030cb407ac5452d9a76231f8449a3a4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 06:40:09.932158 21502 raft_consensus.cc:348] T 7c26f1a96a3047aea5359bfd06dc311f P f030cb407ac5452d9a76231f8449a3a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:09.932251 21502 raft_consensus.cc:374] T 7c26f1a96a3047aea5359bfd06dc311f P f030cb407ac5452d9a76231f8449a3a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:09.932272 21502 raft_consensus.cc:729] T 7c26f1a96a3047aea5359bfd06dc311f P f030cb407ac5452d9a76231f8449a3a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f030cb407ac5452d9a76231f8449a3a4, State: Initialized, Role: FOLLOWER
I20260928 06:40:09.932312 21502 consensus_queue.cc:260] T 7c26f1a96a3047aea5359bfd06dc311f P f030cb407ac5452d9a76231f8449a3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:09.932391 21498 tablet_bootstrap.cc:492] T 7c26f1a96a3047aea5359bfd06dc311f P 5a402cacef4e43ca90ed68ac5f334659: No bootstrap required, opened a new log
I20260928 06:40:09.932428 21502 ts_tablet_manager.cc:1428] T 7c26f1a96a3047aea5359bfd06dc311f P f030cb407ac5452d9a76231f8449a3a4: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260928 06:40:09.932436 21498 ts_tablet_manager.cc:1397] T 7c26f1a96a3047aea5359bfd06dc311f P 5a402cacef4e43ca90ed68ac5f334659: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:09.932487 21499 tablet_bootstrap.cc:492] T 7c26f1a96a3047aea5359bfd06dc311f P 20a3dd6652e74d1b8d6890637a957a51: No bootstrap required, opened a new log
I20260928 06:40:09.932534 21499 ts_tablet_manager.cc:1397] T 7c26f1a96a3047aea5359bfd06dc311f P 20a3dd6652e74d1b8d6890637a957a51: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:09.932551 21506 tablet_replica.cc:439] T 7c26f1a96a3047aea5359bfd06dc311f P f030cb407ac5452d9a76231f8449a3a4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } } }
I20260928 06:40:09.932570 21498 raft_consensus.cc:348] T 7c26f1a96a3047aea5359bfd06dc311f P 5a402cacef4e43ca90ed68ac5f334659 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:09.932626 21506 tablet_replica.cc:442] T 7c26f1a96a3047aea5359bfd06dc311f P f030cb407ac5452d9a76231f8449a3a4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:40:09.932646 21498 raft_consensus.cc:374] T 7c26f1a96a3047aea5359bfd06dc311f P 5a402cacef4e43ca90ed68ac5f334659 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:09.932665 21498 raft_consensus.cc:729] T 7c26f1a96a3047aea5359bfd06dc311f P 5a402cacef4e43ca90ed68ac5f334659 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a402cacef4e43ca90ed68ac5f334659, State: Initialized, Role: FOLLOWER
I20260928 06:40:09.932673 21499 raft_consensus.cc:348] T 7c26f1a96a3047aea5359bfd06dc311f P 20a3dd6652e74d1b8d6890637a957a51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:09.932698 21498 consensus_queue.cc:260] T 7c26f1a96a3047aea5359bfd06dc311f P 5a402cacef4e43ca90ed68ac5f334659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:09.932734 21499 raft_consensus.cc:374] T 7c26f1a96a3047aea5359bfd06dc311f P 20a3dd6652e74d1b8d6890637a957a51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:09.932760 21499 raft_consensus.cc:729] T 7c26f1a96a3047aea5359bfd06dc311f P 20a3dd6652e74d1b8d6890637a957a51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a3dd6652e74d1b8d6890637a957a51, State: Initialized, Role: FOLLOWER
I20260928 06:40:09.932794 21498 ts_tablet_manager.cc:1428] T 7c26f1a96a3047aea5359bfd06dc311f P 5a402cacef4e43ca90ed68ac5f334659: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:09.932798 21499 consensus_queue.cc:260] T 7c26f1a96a3047aea5359bfd06dc311f P 20a3dd6652e74d1b8d6890637a957a51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:09.932893 21499 ts_tablet_manager.cc:1428] T 7c26f1a96a3047aea5359bfd06dc311f P 20a3dd6652e74d1b8d6890637a957a51: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:09.933023 21504 tablet_replica.cc:439] T 7c26f1a96a3047aea5359bfd06dc311f P 5a402cacef4e43ca90ed68ac5f334659: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } } }
I20260928 06:40:09.933107 21504 tablet_replica.cc:442] T 7c26f1a96a3047aea5359bfd06dc311f P 5a402cacef4e43ca90ed68ac5f334659: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:40:09.933262 21505 tablet_replica.cc:439] T 7c26f1a96a3047aea5359bfd06dc311f P 20a3dd6652e74d1b8d6890637a957a51: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } } }
I20260928 06:40:09.933328 21505 tablet_replica.cc:442] T 7c26f1a96a3047aea5359bfd06dc311f P 20a3dd6652e74d1b8d6890637a957a51: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:40:09.947985 21504 raft_consensus.cc:482] T 7c26f1a96a3047aea5359bfd06dc311f P 5a402cacef4e43ca90ed68ac5f334659 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:40:09.948060 21504 raft_consensus.cc:504] T 7c26f1a96a3047aea5359bfd06dc311f P 5a402cacef4e43ca90ed68ac5f334659 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:09.948367 21504 leader_election.cc:288] T 7c26f1a96a3047aea5359bfd06dc311f P 5a402cacef4e43ca90ed68ac5f334659 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f030cb407ac5452d9a76231f8449a3a4 (127.18.211.66:46747), 20a3dd6652e74d1b8d6890637a957a51 (127.18.211.67:34267)
I20260928 06:40:09.952023 21494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:09.948566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:56287 (local address 127.18.211.66:46747)
0928 06:40:09.948616 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:09.948618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:09.948627 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:40:09.948652 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:09.948653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:09.948662 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:09.948717 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:09.948917 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.949658 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.951117 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.951660 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.951734 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:09.951742 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:09.951917 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 06:40:09.951952 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:40:09.952023 21497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:09.948666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:40923 (local address 127.18.211.67:34267)
0928 06:40:09.948710 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:09.948712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:09.948804 (+ 92us) server_negotiation.cc:407] Connection header received
0928 06:40:09.948869 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:09.948869 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:09.948877 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:09.948927 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:09.949021 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.949702 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.951114 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.951661 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.951734 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:09.951742 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:09.951917 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 06:40:09.951952 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:40:09.952060 21091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:09.948500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:46747 (local address 127.18.211.65:56287)
0928 06:40:09.948544 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:09.948555 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:09.948594 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:09.948772 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:09.948773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:09.948777 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:09.948845 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:09.948847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.949752 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.949753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:09.951003 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:09.951006 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.951743 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.951745 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:09.951839 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:09.951841 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:40:09.951895 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 06:40:09.951952 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":8}
I20260928 06:40:09.952071 21519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:09.948594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:34267 (local address 127.18.211.65:40923)
0928 06:40:09.948754 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:09.948765 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:09.948815 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:09.948929 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:09.948929 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:09.948931 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:09.948977 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:09.948978 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.949784 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.949785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:09.951003 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:09.951006 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.951769 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.951770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:09.951837 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:09.951841 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:40:09.951895 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 06:40:09.951952 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":49,"thread_start_us":107,"threads_started":1}
I20260928 06:40:09.952364 21432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c26f1a96a3047aea5359bfd06dc311f" candidate_uuid: "5a402cacef4e43ca90ed68ac5f334659" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a3dd6652e74d1b8d6890637a957a51" is_pre_election: true
I20260928 06:40:09.952445 21432 raft_consensus.cc:2454] T 7c26f1a96a3047aea5359bfd06dc311f P 20a3dd6652e74d1b8d6890637a957a51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a402cacef4e43ca90ed68ac5f334659 in term 0.
I20260928 06:40:09.952591 21292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c26f1a96a3047aea5359bfd06dc311f" candidate_uuid: "5a402cacef4e43ca90ed68ac5f334659" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f030cb407ac5452d9a76231f8449a3a4" is_pre_election: true
I20260928 06:40:09.952747 21292 raft_consensus.cc:2454] T 7c26f1a96a3047aea5359bfd06dc311f P f030cb407ac5452d9a76231f8449a3a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a402cacef4e43ca90ed68ac5f334659 in term 0.
I20260928 06:40:09.952929 21084 leader_election.cc:302] T 7c26f1a96a3047aea5359bfd06dc311f P 5a402cacef4e43ca90ed68ac5f334659 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a402cacef4e43ca90ed68ac5f334659, f030cb407ac5452d9a76231f8449a3a4; no voters:
I20260928 06:40:09.953070 21504 raft_consensus.cc:2781] T 7c26f1a96a3047aea5359bfd06dc311f P 5a402cacef4e43ca90ed68ac5f334659 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:40:09.953121 21504 raft_consensus.cc:482] T 7c26f1a96a3047aea5359bfd06dc311f P 5a402cacef4e43ca90ed68ac5f334659 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:40:09.953136 21504 raft_consensus.cc:3037] T 7c26f1a96a3047aea5359bfd06dc311f P 5a402cacef4e43ca90ed68ac5f334659 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:09.954031 21504 raft_consensus.cc:504] T 7c26f1a96a3047aea5359bfd06dc311f P 5a402cacef4e43ca90ed68ac5f334659 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:09.954138 21504 leader_election.cc:288] T 7c26f1a96a3047aea5359bfd06dc311f P 5a402cacef4e43ca90ed68ac5f334659 [CANDIDATE]: Term 1 election: Requested vote from peers f030cb407ac5452d9a76231f8449a3a4 (127.18.211.66:46747), 20a3dd6652e74d1b8d6890637a957a51 (127.18.211.67:34267)
I20260928 06:40:09.954288 21292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c26f1a96a3047aea5359bfd06dc311f" candidate_uuid: "5a402cacef4e43ca90ed68ac5f334659" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f030cb407ac5452d9a76231f8449a3a4"
I20260928 06:40:09.954349 21292 raft_consensus.cc:3037] T 7c26f1a96a3047aea5359bfd06dc311f P f030cb407ac5452d9a76231f8449a3a4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:09.954329 21432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c26f1a96a3047aea5359bfd06dc311f" candidate_uuid: "5a402cacef4e43ca90ed68ac5f334659" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a3dd6652e74d1b8d6890637a957a51"
I20260928 06:40:09.954406 21432 raft_consensus.cc:3037] T 7c26f1a96a3047aea5359bfd06dc311f P 20a3dd6652e74d1b8d6890637a957a51 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:09.954921 21292 raft_consensus.cc:2454] T 7c26f1a96a3047aea5359bfd06dc311f P f030cb407ac5452d9a76231f8449a3a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a402cacef4e43ca90ed68ac5f334659 in term 1.
I20260928 06:40:09.955035 21432 raft_consensus.cc:2454] T 7c26f1a96a3047aea5359bfd06dc311f P 20a3dd6652e74d1b8d6890637a957a51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a402cacef4e43ca90ed68ac5f334659 in term 1.
I20260928 06:40:09.955168 21084 leader_election.cc:302] T 7c26f1a96a3047aea5359bfd06dc311f P 5a402cacef4e43ca90ed68ac5f334659 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a402cacef4e43ca90ed68ac5f334659, f030cb407ac5452d9a76231f8449a3a4; no voters:
I20260928 06:40:09.955251 21504 raft_consensus.cc:2781] T 7c26f1a96a3047aea5359bfd06dc311f P 5a402cacef4e43ca90ed68ac5f334659 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:09.955390 21504 raft_consensus.cc:686] T 7c26f1a96a3047aea5359bfd06dc311f P 5a402cacef4e43ca90ed68ac5f334659 [term 1 LEADER]: Becoming Leader. State: Replica: 5a402cacef4e43ca90ed68ac5f334659, State: Running, Role: LEADER
I20260928 06:40:09.955475 21504 consensus_queue.cc:237] T 7c26f1a96a3047aea5359bfd06dc311f P 5a402cacef4e43ca90ed68ac5f334659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:09.955894 21520 tablet_replica.cc:439] T 7c26f1a96a3047aea5359bfd06dc311f P 5a402cacef4e43ca90ed68ac5f334659: TxnStatusTablet state changed. Reason: New leader 5a402cacef4e43ca90ed68ac5f334659. Latest consensus state: current_term: 1 leader_uuid: "5a402cacef4e43ca90ed68ac5f334659" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } } }
I20260928 06:40:09.955977 21520 tablet_replica.cc:442] T 7c26f1a96a3047aea5359bfd06dc311f P 5a402cacef4e43ca90ed68ac5f334659: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:40:09.956173 20995 catalog_manager.cc:5495] T 7c26f1a96a3047aea5359bfd06dc311f P 5a402cacef4e43ca90ed68ac5f334659 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a402cacef4e43ca90ed68ac5f334659 (127.18.211.65). New cstate: current_term: 1 leader_uuid: "5a402cacef4e43ca90ed68ac5f334659" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } health_report { overall_health: UNKNOWN } } }
I20260928 06:40:09.956427 21522 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 06:40:09.956528 21522 txn_status_manager.cc:929] Loading transaction status metadata into memory...
W20260928 06:40:09.958292 21199 tablet.cc:2369] T 7c26f1a96a3047aea5359bfd06dc311f P 5a402cacef4e43ca90ed68ac5f334659: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:40:09.963904 21515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:09.961221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:41567 (local address 127.0.0.1:49692)
0928 06:40:09.961290 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:09.961303 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:09.961356 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:09.961517 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:09.961517 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:09.961518 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:40:09.961561 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:09.961563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.962419 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.962420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:09.962987 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:09.962989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.963448 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.963449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:09.963502 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:09.963518 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:40:09.963786 (+ 268us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:40:09.963790 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:40:09.963813 (+ 23us) client_negotiation.cc:240] Negotiation successful
0928 06:40:09.963848 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 06:40:09.963914 21496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:09.961322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49692 (local address 127.18.211.65:41567)
0928 06:40:09.961381 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:09.961383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:09.961392 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:40:09.961422 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:09.961423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:09.961443 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:09.961515 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:40:09.961603 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.962352 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.963064 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.963378 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.963447 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:09.963600 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:40:09.963718 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:40:09.963807 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:09.963823 (+ 16us) server_negotiation.cc:299] Negotiation successful
0928 06:40:09.963846 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 06:40:09.965862 21292 raft_consensus.cc:1261] T 7c26f1a96a3047aea5359bfd06dc311f P f030cb407ac5452d9a76231f8449a3a4 [term 1 FOLLOWER]: Refusing update from remote peer 5a402cacef4e43ca90ed68ac5f334659: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:40:09.965865 21432 raft_consensus.cc:1261] T 7c26f1a96a3047aea5359bfd06dc311f P 20a3dd6652e74d1b8d6890637a957a51 [term 1 FOLLOWER]: Refusing update from remote peer 5a402cacef4e43ca90ed68ac5f334659: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:40:09.966161 21520 consensus_queue.cc:1035] T 7c26f1a96a3047aea5359bfd06dc311f P 5a402cacef4e43ca90ed68ac5f334659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:40:09.966275 21504 consensus_queue.cc:1035] T 7c26f1a96a3047aea5359bfd06dc311f P 5a402cacef4e43ca90ed68ac5f334659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:40:09.967448 21506 tablet_replica.cc:439] T 7c26f1a96a3047aea5359bfd06dc311f P f030cb407ac5452d9a76231f8449a3a4: TxnStatusTablet state changed. Reason: New leader 5a402cacef4e43ca90ed68ac5f334659. Latest consensus state: current_term: 1 leader_uuid: "5a402cacef4e43ca90ed68ac5f334659" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } } }
I20260928 06:40:09.967535 21506 tablet_replica.cc:442] T 7c26f1a96a3047aea5359bfd06dc311f P f030cb407ac5452d9a76231f8449a3a4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:40:09.967535 21505 tablet_replica.cc:439] T 7c26f1a96a3047aea5359bfd06dc311f P 20a3dd6652e74d1b8d6890637a957a51: TxnStatusTablet state changed. Reason: New leader 5a402cacef4e43ca90ed68ac5f334659. Latest consensus state: current_term: 1 leader_uuid: "5a402cacef4e43ca90ed68ac5f334659" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } } }
I20260928 06:40:09.967635 21505 tablet_replica.cc:442] T 7c26f1a96a3047aea5359bfd06dc311f P 20a3dd6652e74d1b8d6890637a957a51: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:40:09.968657 21520 tablet_replica.cc:439] T 7c26f1a96a3047aea5359bfd06dc311f P 5a402cacef4e43ca90ed68ac5f334659: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a402cacef4e43ca90ed68ac5f334659" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } } }
I20260928 06:40:09.968731 21520 tablet_replica.cc:442] T 7c26f1a96a3047aea5359bfd06dc311f P 5a402cacef4e43ca90ed68ac5f334659: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:40:09.968801 21525 mvcc.cc:205] Tried to move back new op lower bound from 7334205890417790976 to 7334205890378317824. Current Snapshot: MvccSnapshot[applied={T|T < 7334205890417790976}]
I20260928 06:40:09.968786 21506 tablet_replica.cc:439] T 7c26f1a96a3047aea5359bfd06dc311f P f030cb407ac5452d9a76231f8449a3a4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a402cacef4e43ca90ed68ac5f334659" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } } }
I20260928 06:40:09.968859 21506 tablet_replica.cc:442] T 7c26f1a96a3047aea5359bfd06dc311f P f030cb407ac5452d9a76231f8449a3a4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:40:09.969017 21527 mvcc.cc:205] Tried to move back new op lower bound from 7334205890417790976 to 7334205890378317824. Current Snapshot: MvccSnapshot[applied={T|T < 7334205890417790976}]
I20260928 06:40:09.969718 21504 tablet_replica.cc:439] T 7c26f1a96a3047aea5359bfd06dc311f P 5a402cacef4e43ca90ed68ac5f334659: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a402cacef4e43ca90ed68ac5f334659" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } } }
I20260928 06:40:09.969807 21504 tablet_replica.cc:442] T 7c26f1a96a3047aea5359bfd06dc311f P 5a402cacef4e43ca90ed68ac5f334659: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:40:09.970311 21505 tablet_replica.cc:439] T 7c26f1a96a3047aea5359bfd06dc311f P 20a3dd6652e74d1b8d6890637a957a51: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a402cacef4e43ca90ed68ac5f334659" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } } }
I20260928 06:40:09.970386 21505 tablet_replica.cc:442] T 7c26f1a96a3047aea5359bfd06dc311f P 20a3dd6652e74d1b8d6890637a957a51: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:40:09.970455 21526 mvcc.cc:205] Tried to move back new op lower bound from 7334205890417790976 to 7334205890378317824. Current Snapshot: MvccSnapshot[applied={T|T < 7334205890417790976}]
I20260928 06:40:09.973793 21494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:09.971268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54902 (local address 127.18.211.66:46747)
0928 06:40:09.971309 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:09.971310 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:09.971321 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:40:09.971392 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:09.971393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:09.971410 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:09.971474 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:40:09.971598 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.972352 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.973053 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.973348 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.973411 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:09.973506 (+ 95us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:40:09.973618 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:40:09.973681 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:09.973697 (+ 16us) server_negotiation.cc:299] Negotiation successful
0928 06:40:09.973716 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:40:09.978513 21496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:09.976131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:53367 (local address 127.18.211.65:41567)
0928 06:40:09.976205 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:09.976207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:09.976216 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:40:09.976236 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:09.976237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:09.976247 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:09.976301 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:40:09.976433 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.976866 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.977943 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.978116 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.978163 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:09.978338 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:40:09.978361 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:40:09.978394 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:09.978450 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 06:40:09.978465 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 06:40:09.978570 21344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:09.975945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:41567 (local address 127.18.211.66:53367)
0928 06:40:09.976009 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:09.976020 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:09.976062 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:09.976314 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:09.976315 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:09.976316 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:40:09.976369 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:09.976370 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.976924 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.976925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:09.977859 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:09.977861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.978179 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.978180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:09.978239 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:09.978254 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:40:09.978400 (+ 146us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:40:09.978402 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:40:09.978447 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 06:40:09.978486 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 06:40:09.980706 21432 raft_consensus.cc:1261] T 9ca657f7b26d4bfba92974472491a794 P 20a3dd6652e74d1b8d6890637a957a51 [term 1 FOLLOWER]: Refusing update from remote peer f030cb407ac5452d9a76231f8449a3a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:40:09.980789 21152 raft_consensus.cc:1261] T 9ca657f7b26d4bfba92974472491a794 P 5a402cacef4e43ca90ed68ac5f334659 [term 1 FOLLOWER]: Refusing update from remote peer f030cb407ac5452d9a76231f8449a3a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:40:09.980926 21506 consensus_queue.cc:1035] T 9ca657f7b26d4bfba92974472491a794 P f030cb407ac5452d9a76231f8449a3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:40:09.981088 21508 consensus_queue.cc:1035] T 9ca657f7b26d4bfba92974472491a794 P f030cb407ac5452d9a76231f8449a3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:40:09.989781 21540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:09.986899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:46747 (local address 127.18.211.65:35789)
0928 06:40:09.987024 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:09.987034 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:09.987126 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:09.987334 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:09.987334 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:09.987336 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:40:09.987403 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:09.987404 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.988272 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.988272 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:09.988886 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:09.988889 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.989370 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.989371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:09.989435 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:09.989449 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:40:09.989597 (+ 148us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:40:09.989599 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:40:09.989638 (+ 39us) client_negotiation.cc:240] Negotiation successful
0928 06:40:09.989683 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":66,"threads_started":1}
I20260928 06:40:09.989764 21494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:09.986920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:35789 (local address 127.18.211.66:46747)
0928 06:40:09.986990 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:09.986992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:09.987105 (+ 113us) server_negotiation.cc:407] Connection header received
0928 06:40:09.987197 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:09.987198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:09.987208 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:09.987271 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:40:09.987457 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.988193 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.989029 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:09.989306 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:09.989362 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:09.989513 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:40:09.989547 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:40:09.989589 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:09.989682 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 06:40:09.989699 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 06:40:09.996310 20994 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:40:09.996414 20994 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:40:09.997223 20994 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b309eba9ace4bb4a324ccd7217e4227: Sending DeleteTablet for 3 replicas of tablet 9ca657f7b26d4bfba92974472491a794
I20260928 06:40:09.997517 21272 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ca657f7b26d4bfba92974472491a794 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:09 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:54890
I20260928 06:40:09.997597 21412 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ca657f7b26d4bfba92974472491a794 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:09 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:42698
I20260928 06:40:09.997735 21544 tablet_replica.cc:330] T 9ca657f7b26d4bfba92974472491a794 P f030cb407ac5452d9a76231f8449a3a4: stopping tablet replica
I20260928 06:40:09.997822 21544 raft_consensus.cc:2229] T 9ca657f7b26d4bfba92974472491a794 P f030cb407ac5452d9a76231f8449a3a4 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:40:09.997911 21544 raft_consensus.cc:2258] T 9ca657f7b26d4bfba92974472491a794 P f030cb407ac5452d9a76231f8449a3a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:09.998093 21132 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ca657f7b26d4bfba92974472491a794 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:09 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:49684
I20260928 06:40:09.998363 21544 ts_tablet_manager.cc:1905] T 9ca657f7b26d4bfba92974472491a794 P f030cb407ac5452d9a76231f8449a3a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:09.998621 20994 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53618:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:40:09.998760 20994 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:40:10.000121 21544 ts_tablet_manager.cc:1918] T 9ca657f7b26d4bfba92974472491a794 P f030cb407ac5452d9a76231f8449a3a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:40:10.000204 21544 log.cc:1196] T 9ca657f7b26d4bfba92974472491a794 P f030cb407ac5452d9a76231f8449a3a4: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-1/wal/wals/9ca657f7b26d4bfba92974472491a794
I20260928 06:40:10.000244 21272 tablet_service.cc:1467] Processing CreateTablet for tablet 203a82d7f8d344fe8f14362d98a70028 (DEFAULT_TABLE table=test-table [id=5da127491539457c97d428525267371a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:10.000352 21272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 203a82d7f8d344fe8f14362d98a70028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:10.000447 21544 ts_tablet_manager.cc:1939] T 9ca657f7b26d4bfba92974472491a794 P f030cb407ac5452d9a76231f8449a3a4: Deleting consensus metadata
I20260928 06:40:10.000815 20980 catalog_manager.cc:4841] TS f030cb407ac5452d9a76231f8449a3a4 (127.18.211.66:46747): tablet 9ca657f7b26d4bfba92974472491a794 (table test-table [id=eafea5d540ad46f6827a834a63e47376]) successfully deleted
I20260928 06:40:10.000978 21546 tablet_replica.cc:330] T 9ca657f7b26d4bfba92974472491a794 P 5a402cacef4e43ca90ed68ac5f334659: stopping tablet replica
I20260928 06:40:10.001075 21546 raft_consensus.cc:2229] T 9ca657f7b26d4bfba92974472491a794 P 5a402cacef4e43ca90ed68ac5f334659 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:40:10.001122 21546 raft_consensus.cc:2258] T 9ca657f7b26d4bfba92974472491a794 P 5a402cacef4e43ca90ed68ac5f334659 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:10.001456 21502 tablet_bootstrap.cc:492] T 203a82d7f8d344fe8f14362d98a70028 P f030cb407ac5452d9a76231f8449a3a4: Bootstrap starting.
I20260928 06:40:10.001535 21546 ts_tablet_manager.cc:1905] T 9ca657f7b26d4bfba92974472491a794 P 5a402cacef4e43ca90ed68ac5f334659: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:10.001833 21502 tablet_bootstrap.cc:654] T 203a82d7f8d344fe8f14362d98a70028 P f030cb407ac5452d9a76231f8449a3a4: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:10.002285 21412 tablet_service.cc:1467] Processing CreateTablet for tablet 203a82d7f8d344fe8f14362d98a70028 (DEFAULT_TABLE table=test-table [id=5da127491539457c97d428525267371a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:10.002385 21412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 203a82d7f8d344fe8f14362d98a70028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:10.003458 21546 ts_tablet_manager.cc:1918] T 9ca657f7b26d4bfba92974472491a794 P 5a402cacef4e43ca90ed68ac5f334659: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:40:10.003516 21546 log.cc:1196] T 9ca657f7b26d4bfba92974472491a794 P 5a402cacef4e43ca90ed68ac5f334659: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-0/wal/wals/9ca657f7b26d4bfba92974472491a794
I20260928 06:40:10.003670 21499 tablet_bootstrap.cc:492] T 203a82d7f8d344fe8f14362d98a70028 P 20a3dd6652e74d1b8d6890637a957a51: Bootstrap starting.
I20260928 06:40:10.003742 21546 ts_tablet_manager.cc:1939] T 9ca657f7b26d4bfba92974472491a794 P 5a402cacef4e43ca90ed68ac5f334659: Deleting consensus metadata
I20260928 06:40:10.004038 21499 tablet_bootstrap.cc:654] T 203a82d7f8d344fe8f14362d98a70028 P 20a3dd6652e74d1b8d6890637a957a51: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:10.004667 21499 tablet_bootstrap.cc:492] T 203a82d7f8d344fe8f14362d98a70028 P 20a3dd6652e74d1b8d6890637a957a51: No bootstrap required, opened a new log
I20260928 06:40:10.004720 21499 ts_tablet_manager.cc:1397] T 203a82d7f8d344fe8f14362d98a70028 P 20a3dd6652e74d1b8d6890637a957a51: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:10.004853 21499 raft_consensus.cc:348] T 203a82d7f8d344fe8f14362d98a70028 P 20a3dd6652e74d1b8d6890637a957a51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:10.004884 20980 catalog_manager.cc:4841] TS 5a402cacef4e43ca90ed68ac5f334659 (127.18.211.65:41567): tablet 9ca657f7b26d4bfba92974472491a794 (table test-table [id=eafea5d540ad46f6827a834a63e47376]) successfully deleted
I20260928 06:40:10.004923 21499 raft_consensus.cc:374] T 203a82d7f8d344fe8f14362d98a70028 P 20a3dd6652e74d1b8d6890637a957a51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:10.004944 21499 raft_consensus.cc:729] T 203a82d7f8d344fe8f14362d98a70028 P 20a3dd6652e74d1b8d6890637a957a51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a3dd6652e74d1b8d6890637a957a51, State: Initialized, Role: FOLLOWER
I20260928 06:40:10.004985 21499 consensus_queue.cc:260] T 203a82d7f8d344fe8f14362d98a70028 P 20a3dd6652e74d1b8d6890637a957a51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:10.004983 21132 tablet_service.cc:1467] Processing CreateTablet for tablet 203a82d7f8d344fe8f14362d98a70028 (DEFAULT_TABLE table=test-table [id=5da127491539457c97d428525267371a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:10.005065 21499 ts_tablet_manager.cc:1428] T 203a82d7f8d344fe8f14362d98a70028 P 20a3dd6652e74d1b8d6890637a957a51: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:10.005072 21132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 203a82d7f8d344fe8f14362d98a70028. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:10.006171 21498 tablet_bootstrap.cc:492] T 203a82d7f8d344fe8f14362d98a70028 P 5a402cacef4e43ca90ed68ac5f334659: Bootstrap starting.
I20260928 06:40:10.006558 21498 tablet_bootstrap.cc:654] T 203a82d7f8d344fe8f14362d98a70028 P 5a402cacef4e43ca90ed68ac5f334659: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:10.007205 21498 tablet_bootstrap.cc:492] T 203a82d7f8d344fe8f14362d98a70028 P 5a402cacef4e43ca90ed68ac5f334659: No bootstrap required, opened a new log
I20260928 06:40:10.007251 21498 ts_tablet_manager.cc:1397] T 203a82d7f8d344fe8f14362d98a70028 P 5a402cacef4e43ca90ed68ac5f334659: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:10.007373 21498 raft_consensus.cc:348] T 203a82d7f8d344fe8f14362d98a70028 P 5a402cacef4e43ca90ed68ac5f334659 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:10.007437 21498 raft_consensus.cc:374] T 203a82d7f8d344fe8f14362d98a70028 P 5a402cacef4e43ca90ed68ac5f334659 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:10.007457 21498 raft_consensus.cc:729] T 203a82d7f8d344fe8f14362d98a70028 P 5a402cacef4e43ca90ed68ac5f334659 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a402cacef4e43ca90ed68ac5f334659, State: Initialized, Role: FOLLOWER
I20260928 06:40:10.007494 21498 consensus_queue.cc:260] T 203a82d7f8d344fe8f14362d98a70028 P 5a402cacef4e43ca90ed68ac5f334659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:10.007581 21498 ts_tablet_manager.cc:1428] T 203a82d7f8d344fe8f14362d98a70028 P 5a402cacef4e43ca90ed68ac5f334659: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:10.007995 21502 tablet_bootstrap.cc:492] T 203a82d7f8d344fe8f14362d98a70028 P f030cb407ac5452d9a76231f8449a3a4: No bootstrap required, opened a new log
I20260928 06:40:10.008044 21502 ts_tablet_manager.cc:1397] T 203a82d7f8d344fe8f14362d98a70028 P f030cb407ac5452d9a76231f8449a3a4: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260928 06:40:10.008183 21502 raft_consensus.cc:348] T 203a82d7f8d344fe8f14362d98a70028 P f030cb407ac5452d9a76231f8449a3a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:10.008239 21502 raft_consensus.cc:374] T 203a82d7f8d344fe8f14362d98a70028 P f030cb407ac5452d9a76231f8449a3a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:10.008255 21502 raft_consensus.cc:729] T 203a82d7f8d344fe8f14362d98a70028 P f030cb407ac5452d9a76231f8449a3a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f030cb407ac5452d9a76231f8449a3a4, State: Initialized, Role: FOLLOWER
I20260928 06:40:10.008296 21502 consensus_queue.cc:260] T 203a82d7f8d344fe8f14362d98a70028 P f030cb407ac5452d9a76231f8449a3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:10.008379 21502 ts_tablet_manager.cc:1428] T 203a82d7f8d344fe8f14362d98a70028 P f030cb407ac5452d9a76231f8449a3a4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:10.008672 21545 tablet_replica.cc:330] T 9ca657f7b26d4bfba92974472491a794 P 20a3dd6652e74d1b8d6890637a957a51: stopping tablet replica
I20260928 06:40:10.008827 21545 raft_consensus.cc:2229] T 9ca657f7b26d4bfba92974472491a794 P 20a3dd6652e74d1b8d6890637a957a51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:40:10.008924 21545 raft_consensus.cc:2258] T 9ca657f7b26d4bfba92974472491a794 P 20a3dd6652e74d1b8d6890637a957a51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:10.009552 21545 ts_tablet_manager.cc:1905] T 9ca657f7b26d4bfba92974472491a794 P 20a3dd6652e74d1b8d6890637a957a51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:10.011370 21545 ts_tablet_manager.cc:1918] T 9ca657f7b26d4bfba92974472491a794 P 20a3dd6652e74d1b8d6890637a957a51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:40:10.011508 21545 log.cc:1196] T 9ca657f7b26d4bfba92974472491a794 P 20a3dd6652e74d1b8d6890637a957a51: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-2/wal/wals/9ca657f7b26d4bfba92974472491a794
I20260928 06:40:10.011996 21545 ts_tablet_manager.cc:1939] T 9ca657f7b26d4bfba92974472491a794 P 20a3dd6652e74d1b8d6890637a957a51: Deleting consensus metadata
I20260928 06:40:10.012499 20980 catalog_manager.cc:4841] TS 20a3dd6652e74d1b8d6890637a957a51 (127.18.211.67:34267): tablet 9ca657f7b26d4bfba92974472491a794 (table test-table [id=eafea5d540ad46f6827a834a63e47376]) successfully deleted
W20260928 06:40:10.022315 21479 tablet.cc:2369] T 203a82d7f8d344fe8f14362d98a70028 P 20a3dd6652e74d1b8d6890637a957a51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:40:10.132483 21541 raft_consensus.cc:482] T 203a82d7f8d344fe8f14362d98a70028 P f030cb407ac5452d9a76231f8449a3a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:40:10.132581 21541 raft_consensus.cc:504] T 203a82d7f8d344fe8f14362d98a70028 P f030cb407ac5452d9a76231f8449a3a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:10.132711 21541 leader_election.cc:288] T 203a82d7f8d344fe8f14362d98a70028 P f030cb407ac5452d9a76231f8449a3a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a402cacef4e43ca90ed68ac5f334659 (127.18.211.65:41567), 20a3dd6652e74d1b8d6890637a957a51 (127.18.211.67:34267)
I20260928 06:40:10.132963 21432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "203a82d7f8d344fe8f14362d98a70028" candidate_uuid: "f030cb407ac5452d9a76231f8449a3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a3dd6652e74d1b8d6890637a957a51" is_pre_election: true
I20260928 06:40:10.132959 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "203a82d7f8d344fe8f14362d98a70028" candidate_uuid: "f030cb407ac5452d9a76231f8449a3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a402cacef4e43ca90ed68ac5f334659" is_pre_election: true
I20260928 06:40:10.133059 21432 raft_consensus.cc:2454] T 203a82d7f8d344fe8f14362d98a70028 P 20a3dd6652e74d1b8d6890637a957a51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f030cb407ac5452d9a76231f8449a3a4 in term 0.
I20260928 06:40:10.133070 21152 raft_consensus.cc:2454] T 203a82d7f8d344fe8f14362d98a70028 P 5a402cacef4e43ca90ed68ac5f334659 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f030cb407ac5452d9a76231f8449a3a4 in term 0.
I20260928 06:40:10.133213 21224 leader_election.cc:302] T 203a82d7f8d344fe8f14362d98a70028 P f030cb407ac5452d9a76231f8449a3a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a3dd6652e74d1b8d6890637a957a51, f030cb407ac5452d9a76231f8449a3a4; no voters:
I20260928 06:40:10.133338 21541 raft_consensus.cc:2781] T 203a82d7f8d344fe8f14362d98a70028 P f030cb407ac5452d9a76231f8449a3a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:40:10.133380 21541 raft_consensus.cc:482] T 203a82d7f8d344fe8f14362d98a70028 P f030cb407ac5452d9a76231f8449a3a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:40:10.133399 21541 raft_consensus.cc:3037] T 203a82d7f8d344fe8f14362d98a70028 P f030cb407ac5452d9a76231f8449a3a4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:10.134116 21541 raft_consensus.cc:504] T 203a82d7f8d344fe8f14362d98a70028 P f030cb407ac5452d9a76231f8449a3a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:10.134218 21541 leader_election.cc:288] T 203a82d7f8d344fe8f14362d98a70028 P f030cb407ac5452d9a76231f8449a3a4 [CANDIDATE]: Term 1 election: Requested vote from peers 5a402cacef4e43ca90ed68ac5f334659 (127.18.211.65:41567), 20a3dd6652e74d1b8d6890637a957a51 (127.18.211.67:34267)
I20260928 06:40:10.134404 21152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "203a82d7f8d344fe8f14362d98a70028" candidate_uuid: "f030cb407ac5452d9a76231f8449a3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a402cacef4e43ca90ed68ac5f334659"
I20260928 06:40:10.134428 21432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "203a82d7f8d344fe8f14362d98a70028" candidate_uuid: "f030cb407ac5452d9a76231f8449a3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a3dd6652e74d1b8d6890637a957a51"
I20260928 06:40:10.134481 21432 raft_consensus.cc:3037] T 203a82d7f8d344fe8f14362d98a70028 P 20a3dd6652e74d1b8d6890637a957a51 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:10.134481 21152 raft_consensus.cc:3037] T 203a82d7f8d344fe8f14362d98a70028 P 5a402cacef4e43ca90ed68ac5f334659 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:10.135046 21432 raft_consensus.cc:2454] T 203a82d7f8d344fe8f14362d98a70028 P 20a3dd6652e74d1b8d6890637a957a51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f030cb407ac5452d9a76231f8449a3a4 in term 1.
I20260928 06:40:10.135155 21152 raft_consensus.cc:2454] T 203a82d7f8d344fe8f14362d98a70028 P 5a402cacef4e43ca90ed68ac5f334659 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f030cb407ac5452d9a76231f8449a3a4 in term 1.
I20260928 06:40:10.135217 21224 leader_election.cc:302] T 203a82d7f8d344fe8f14362d98a70028 P f030cb407ac5452d9a76231f8449a3a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a3dd6652e74d1b8d6890637a957a51, f030cb407ac5452d9a76231f8449a3a4; no voters:
I20260928 06:40:10.135320 21541 raft_consensus.cc:2781] T 203a82d7f8d344fe8f14362d98a70028 P f030cb407ac5452d9a76231f8449a3a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:10.135377 21541 raft_consensus.cc:686] T 203a82d7f8d344fe8f14362d98a70028 P f030cb407ac5452d9a76231f8449a3a4 [term 1 LEADER]: Becoming Leader. State: Replica: f030cb407ac5452d9a76231f8449a3a4, State: Running, Role: LEADER
I20260928 06:40:10.135433 21541 consensus_queue.cc:237] T 203a82d7f8d344fe8f14362d98a70028 P f030cb407ac5452d9a76231f8449a3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:10.135999 20994 catalog_manager.cc:5495] T 203a82d7f8d344fe8f14362d98a70028 P f030cb407ac5452d9a76231f8449a3a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to f030cb407ac5452d9a76231f8449a3a4 (127.18.211.66). New cstate: current_term: 1 leader_uuid: "f030cb407ac5452d9a76231f8449a3a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } health_report { overall_health: UNKNOWN } } }
I20260928 06:40:10.157124 21432 raft_consensus.cc:1261] T 203a82d7f8d344fe8f14362d98a70028 P 20a3dd6652e74d1b8d6890637a957a51 [term 1 FOLLOWER]: Refusing update from remote peer f030cb407ac5452d9a76231f8449a3a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:40:10.157289 21152 raft_consensus.cc:1261] T 203a82d7f8d344fe8f14362d98a70028 P 5a402cacef4e43ca90ed68ac5f334659 [term 1 FOLLOWER]: Refusing update from remote peer f030cb407ac5452d9a76231f8449a3a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:40:10.157344 21506 consensus_queue.cc:1035] T 203a82d7f8d344fe8f14362d98a70028 P f030cb407ac5452d9a76231f8449a3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:40:10.157496 21541 consensus_queue.cc:1035] T 203a82d7f8d344fe8f14362d98a70028 P f030cb407ac5452d9a76231f8449a3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 06:40:10.161702 21527 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 06:40:11.678354 20995 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:40:11.678454 20995 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53618:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:40:11.679323 20995 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b309eba9ace4bb4a324ccd7217e4227: Sending DeleteTablet for 3 replicas of tablet 203a82d7f8d344fe8f14362d98a70028
I20260928 06:40:11.679560 21132 tablet_service.cc:1514] Processing DeleteTablet for tablet 203a82d7f8d344fe8f14362d98a70028 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:11 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:49684
I20260928 06:40:11.679702 21412 tablet_service.cc:1514] Processing DeleteTablet for tablet 203a82d7f8d344fe8f14362d98a70028 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:11 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:42698
I20260928 06:40:11.679741 21272 tablet_service.cc:1514] Processing DeleteTablet for tablet 203a82d7f8d344fe8f14362d98a70028 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:11 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:54890
I20260928 06:40:11.679955 21578 tablet_replica.cc:330] T 203a82d7f8d344fe8f14362d98a70028 P f030cb407ac5452d9a76231f8449a3a4: stopping tablet replica
I20260928 06:40:11.680009 21578 raft_consensus.cc:2229] T 203a82d7f8d344fe8f14362d98a70028 P f030cb407ac5452d9a76231f8449a3a4 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:40:11.680052 21578 raft_consensus.cc:2258] T 203a82d7f8d344fe8f14362d98a70028 P f030cb407ac5452d9a76231f8449a3a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:11.680141 21576 tablet_replica.cc:330] T 203a82d7f8d344fe8f14362d98a70028 P 5a402cacef4e43ca90ed68ac5f334659: stopping tablet replica
I20260928 06:40:11.680197 21576 raft_consensus.cc:2229] T 203a82d7f8d344fe8f14362d98a70028 P 5a402cacef4e43ca90ed68ac5f334659 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:40:11.680230 21576 raft_consensus.cc:2258] T 203a82d7f8d344fe8f14362d98a70028 P 5a402cacef4e43ca90ed68ac5f334659 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:11.680327 21577 tablet_replica.cc:330] T 203a82d7f8d344fe8f14362d98a70028 P 20a3dd6652e74d1b8d6890637a957a51: stopping tablet replica
I20260928 06:40:11.680383 21577 raft_consensus.cc:2229] T 203a82d7f8d344fe8f14362d98a70028 P 20a3dd6652e74d1b8d6890637a957a51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:40:11.680418 21577 raft_consensus.cc:2258] T 203a82d7f8d344fe8f14362d98a70028 P 20a3dd6652e74d1b8d6890637a957a51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:11.680615 21577 ts_tablet_manager.cc:1905] T 203a82d7f8d344fe8f14362d98a70028 P 20a3dd6652e74d1b8d6890637a957a51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:11.682268 21577 ts_tablet_manager.cc:1918] T 203a82d7f8d344fe8f14362d98a70028 P 20a3dd6652e74d1b8d6890637a957a51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:40:11.682461 21576 ts_tablet_manager.cc:1905] T 203a82d7f8d344fe8f14362d98a70028 P 5a402cacef4e43ca90ed68ac5f334659: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:11.682528 21578 ts_tablet_manager.cc:1905] T 203a82d7f8d344fe8f14362d98a70028 P f030cb407ac5452d9a76231f8449a3a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:11.683969 21576 ts_tablet_manager.cc:1918] T 203a82d7f8d344fe8f14362d98a70028 P 5a402cacef4e43ca90ed68ac5f334659: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:40:11.684027 21576 log.cc:1196] T 203a82d7f8d344fe8f14362d98a70028 P 5a402cacef4e43ca90ed68ac5f334659: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-0/wal/wals/203a82d7f8d344fe8f14362d98a70028
I20260928 06:40:11.684047 21578 ts_tablet_manager.cc:1918] T 203a82d7f8d344fe8f14362d98a70028 P f030cb407ac5452d9a76231f8449a3a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:40:11.684105 21578 log.cc:1196] T 203a82d7f8d344fe8f14362d98a70028 P f030cb407ac5452d9a76231f8449a3a4: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-1/wal/wals/203a82d7f8d344fe8f14362d98a70028
I20260928 06:40:11.684227 21576 ts_tablet_manager.cc:1939] T 203a82d7f8d344fe8f14362d98a70028 P 5a402cacef4e43ca90ed68ac5f334659: Deleting consensus metadata
I20260928 06:40:11.684333 21578 ts_tablet_manager.cc:1939] T 203a82d7f8d344fe8f14362d98a70028 P f030cb407ac5452d9a76231f8449a3a4: Deleting consensus metadata
I20260928 06:40:11.684548 20980 catalog_manager.cc:4841] TS 5a402cacef4e43ca90ed68ac5f334659 (127.18.211.65:41567): tablet 203a82d7f8d344fe8f14362d98a70028 (table test-table [id=5da127491539457c97d428525267371a]) successfully deleted
I20260928 06:40:11.684631 20980 catalog_manager.cc:4841] TS f030cb407ac5452d9a76231f8449a3a4 (127.18.211.66:46747): tablet 203a82d7f8d344fe8f14362d98a70028 (table test-table [id=5da127491539457c97d428525267371a]) successfully deleted
I20260928 06:40:11.685783 21577 log.cc:1196] T 203a82d7f8d344fe8f14362d98a70028 P 20a3dd6652e74d1b8d6890637a957a51: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-2/wal/wals/203a82d7f8d344fe8f14362d98a70028
I20260928 06:40:11.685977 21577 ts_tablet_manager.cc:1939] T 203a82d7f8d344fe8f14362d98a70028 P 20a3dd6652e74d1b8d6890637a957a51: Deleting consensus metadata
I20260928 06:40:11.686223 20980 catalog_manager.cc:4841] TS 20a3dd6652e74d1b8d6890637a957a51 (127.18.211.67:34267): tablet 203a82d7f8d344fe8f14362d98a70028 (table test-table [id=5da127491539457c97d428525267371a]) successfully deleted
Sep 28 06:40:11 dist-test-slave-b28r krb5kdc[20950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577611, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 06:40:11 dist-test-slave-b28r krb5kdc[20950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577611, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:11.705031 21586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:11.697803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53646 (local address 127.18.211.126:37569)
0928 06:40:11.697936 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:11.697937 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:11.697946 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:40:11.697988 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:11.697988 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:11.697999 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:11.698060 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:11.698623 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:11.699225 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:11.700072 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:11.700308 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:11.700360 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:11.702524 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:11.702525 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:11.702526 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:11.702532 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:11.703938 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:11.704281 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:11.704282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:11.704282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:11.704364 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:11.704571 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:11.704571 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:11.704572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:11.704675 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:11.704750 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:11.704888 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 06:40:11.704953 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260928 06:40:11.706501 20995 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:40:11.706655 20995 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:40:11.708024 21132 tablet_service.cc:1467] Processing CreateTablet for tablet dbf3b3abf580453fb14e7fa1acb1b6b2 (DEFAULT_TABLE table=test-table [id=0a68bf3c06c5488d94c29edd10f84882]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:11.708047 21272 tablet_service.cc:1467] Processing CreateTablet for tablet dbf3b3abf580453fb14e7fa1acb1b6b2 (DEFAULT_TABLE table=test-table [id=0a68bf3c06c5488d94c29edd10f84882]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:11.708086 21412 tablet_service.cc:1467] Processing CreateTablet for tablet dbf3b3abf580453fb14e7fa1acb1b6b2 (DEFAULT_TABLE table=test-table [id=0a68bf3c06c5488d94c29edd10f84882]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:11.708158 21272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbf3b3abf580453fb14e7fa1acb1b6b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:11.708158 21132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbf3b3abf580453fb14e7fa1acb1b6b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:11.708230 21412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbf3b3abf580453fb14e7fa1acb1b6b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:11.709230 21587 tablet_bootstrap.cc:492] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 20a3dd6652e74d1b8d6890637a957a51: Bootstrap starting.
I20260928 06:40:11.709589 21589 tablet_bootstrap.cc:492] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 5a402cacef4e43ca90ed68ac5f334659: Bootstrap starting.
I20260928 06:40:11.709656 21587 tablet_bootstrap.cc:654] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 20a3dd6652e74d1b8d6890637a957a51: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:11.709779 21588 tablet_bootstrap.cc:492] T dbf3b3abf580453fb14e7fa1acb1b6b2 P f030cb407ac5452d9a76231f8449a3a4: Bootstrap starting.
I20260928 06:40:11.709928 21589 tablet_bootstrap.cc:654] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 5a402cacef4e43ca90ed68ac5f334659: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:11.710191 21588 tablet_bootstrap.cc:654] T dbf3b3abf580453fb14e7fa1acb1b6b2 P f030cb407ac5452d9a76231f8449a3a4: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:11.710438 21587 tablet_bootstrap.cc:492] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 20a3dd6652e74d1b8d6890637a957a51: No bootstrap required, opened a new log
I20260928 06:40:11.710500 21587 ts_tablet_manager.cc:1397] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 20a3dd6652e74d1b8d6890637a957a51: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:11.710647 21587 raft_consensus.cc:348] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 20a3dd6652e74d1b8d6890637a957a51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:11.710733 21587 raft_consensus.cc:374] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 20a3dd6652e74d1b8d6890637a957a51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:11.710757 21587 raft_consensus.cc:729] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 20a3dd6652e74d1b8d6890637a957a51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a3dd6652e74d1b8d6890637a957a51, State: Initialized, Role: FOLLOWER
I20260928 06:40:11.710808 21588 tablet_bootstrap.cc:492] T dbf3b3abf580453fb14e7fa1acb1b6b2 P f030cb407ac5452d9a76231f8449a3a4: No bootstrap required, opened a new log
I20260928 06:40:11.710815 21587 consensus_queue.cc:260] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 20a3dd6652e74d1b8d6890637a957a51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:11.710861 21588 ts_tablet_manager.cc:1397] T dbf3b3abf580453fb14e7fa1acb1b6b2 P f030cb407ac5452d9a76231f8449a3a4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:11.710999 21587 ts_tablet_manager.cc:1428] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 20a3dd6652e74d1b8d6890637a957a51: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:40:11.710978 21588 raft_consensus.cc:348] T dbf3b3abf580453fb14e7fa1acb1b6b2 P f030cb407ac5452d9a76231f8449a3a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:11.711042 21588 raft_consensus.cc:374] T dbf3b3abf580453fb14e7fa1acb1b6b2 P f030cb407ac5452d9a76231f8449a3a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:11.711130 21588 raft_consensus.cc:729] T dbf3b3abf580453fb14e7fa1acb1b6b2 P f030cb407ac5452d9a76231f8449a3a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f030cb407ac5452d9a76231f8449a3a4, State: Initialized, Role: FOLLOWER
I20260928 06:40:11.711390 21589 tablet_bootstrap.cc:492] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 5a402cacef4e43ca90ed68ac5f334659: No bootstrap required, opened a new log
I20260928 06:40:11.711444 21589 ts_tablet_manager.cc:1397] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 5a402cacef4e43ca90ed68ac5f334659: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:40:11.711268 21588 consensus_queue.cc:260] T dbf3b3abf580453fb14e7fa1acb1b6b2 P f030cb407ac5452d9a76231f8449a3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:11.711557 21589 raft_consensus.cc:348] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 5a402cacef4e43ca90ed68ac5f334659 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:11.711627 21589 raft_consensus.cc:374] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 5a402cacef4e43ca90ed68ac5f334659 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:11.711643 21589 raft_consensus.cc:729] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 5a402cacef4e43ca90ed68ac5f334659 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a402cacef4e43ca90ed68ac5f334659, State: Initialized, Role: FOLLOWER
I20260928 06:40:11.711683 21589 consensus_queue.cc:260] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 5a402cacef4e43ca90ed68ac5f334659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:11.711773 21589 ts_tablet_manager.cc:1428] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 5a402cacef4e43ca90ed68ac5f334659: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:11.711944 21588 ts_tablet_manager.cc:1428] T dbf3b3abf580453fb14e7fa1acb1b6b2 P f030cb407ac5452d9a76231f8449a3a4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:11.936478 21520 raft_consensus.cc:482] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 5a402cacef4e43ca90ed68ac5f334659 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:40:11.936571 21520 raft_consensus.cc:504] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 5a402cacef4e43ca90ed68ac5f334659 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:11.936764 21520 leader_election.cc:288] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 5a402cacef4e43ca90ed68ac5f334659 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f030cb407ac5452d9a76231f8449a3a4 (127.18.211.66:46747), 20a3dd6652e74d1b8d6890637a957a51 (127.18.211.67:34267)
I20260928 06:40:11.936944 21432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbf3b3abf580453fb14e7fa1acb1b6b2" candidate_uuid: "5a402cacef4e43ca90ed68ac5f334659" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a3dd6652e74d1b8d6890637a957a51" is_pre_election: true
I20260928 06:40:11.936934 21292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbf3b3abf580453fb14e7fa1acb1b6b2" candidate_uuid: "5a402cacef4e43ca90ed68ac5f334659" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f030cb407ac5452d9a76231f8449a3a4" is_pre_election: true
I20260928 06:40:11.937029 21432 raft_consensus.cc:2454] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 20a3dd6652e74d1b8d6890637a957a51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a402cacef4e43ca90ed68ac5f334659 in term 0.
I20260928 06:40:11.937033 21292 raft_consensus.cc:2454] T dbf3b3abf580453fb14e7fa1acb1b6b2 P f030cb407ac5452d9a76231f8449a3a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a402cacef4e43ca90ed68ac5f334659 in term 0.
I20260928 06:40:11.937192 21084 leader_election.cc:302] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 5a402cacef4e43ca90ed68ac5f334659 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a402cacef4e43ca90ed68ac5f334659, f030cb407ac5452d9a76231f8449a3a4; no voters:
I20260928 06:40:11.937299 21520 raft_consensus.cc:2781] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 5a402cacef4e43ca90ed68ac5f334659 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:40:11.937351 21520 raft_consensus.cc:482] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 5a402cacef4e43ca90ed68ac5f334659 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:40:11.937366 21520 raft_consensus.cc:3037] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 5a402cacef4e43ca90ed68ac5f334659 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:11.938050 21520 raft_consensus.cc:504] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 5a402cacef4e43ca90ed68ac5f334659 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:11.938162 21520 leader_election.cc:288] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 5a402cacef4e43ca90ed68ac5f334659 [CANDIDATE]: Term 1 election: Requested vote from peers f030cb407ac5452d9a76231f8449a3a4 (127.18.211.66:46747), 20a3dd6652e74d1b8d6890637a957a51 (127.18.211.67:34267)
I20260928 06:40:11.938323 21292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbf3b3abf580453fb14e7fa1acb1b6b2" candidate_uuid: "5a402cacef4e43ca90ed68ac5f334659" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f030cb407ac5452d9a76231f8449a3a4"
I20260928 06:40:11.938403 21292 raft_consensus.cc:3037] T dbf3b3abf580453fb14e7fa1acb1b6b2 P f030cb407ac5452d9a76231f8449a3a4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:11.938395 21432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbf3b3abf580453fb14e7fa1acb1b6b2" candidate_uuid: "5a402cacef4e43ca90ed68ac5f334659" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a3dd6652e74d1b8d6890637a957a51"
I20260928 06:40:11.938482 21432 raft_consensus.cc:3037] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 20a3dd6652e74d1b8d6890637a957a51 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:11.939007 21432 raft_consensus.cc:2454] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 20a3dd6652e74d1b8d6890637a957a51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a402cacef4e43ca90ed68ac5f334659 in term 1.
I20260928 06:40:11.939041 21292 raft_consensus.cc:2454] T dbf3b3abf580453fb14e7fa1acb1b6b2 P f030cb407ac5452d9a76231f8449a3a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a402cacef4e43ca90ed68ac5f334659 in term 1.
I20260928 06:40:11.939183 21084 leader_election.cc:302] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 5a402cacef4e43ca90ed68ac5f334659 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a3dd6652e74d1b8d6890637a957a51, 5a402cacef4e43ca90ed68ac5f334659; no voters:
I20260928 06:40:11.939283 21520 raft_consensus.cc:2781] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 5a402cacef4e43ca90ed68ac5f334659 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:11.939340 21520 raft_consensus.cc:686] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 5a402cacef4e43ca90ed68ac5f334659 [term 1 LEADER]: Becoming Leader. State: Replica: 5a402cacef4e43ca90ed68ac5f334659, State: Running, Role: LEADER
I20260928 06:40:11.939386 21520 consensus_queue.cc:237] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 5a402cacef4e43ca90ed68ac5f334659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:11.939983 20995 catalog_manager.cc:5495] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 5a402cacef4e43ca90ed68ac5f334659 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a402cacef4e43ca90ed68ac5f334659 (127.18.211.65). New cstate: current_term: 1 leader_uuid: "5a402cacef4e43ca90ed68ac5f334659" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } health_report { overall_health: UNKNOWN } } }
I20260928 06:40:11.996147 21596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:11.993779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49706 (local address 127.18.211.65:41567)
0928 06:40:11.993977 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:11.993978 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:11.993987 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:40:11.994006 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:11.994006 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:11.994016 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:11.994080 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:40:11.994242 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:11.994723 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:11.995504 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:11.995712 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:11.995778 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:11.995943 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:40:11.995972 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:40:11.996008 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:11.996052 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 06:40:11.996069 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":102,"threads_started":1}
I20260928 06:40:12.000571 21596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:11.998363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:42699 (local address 127.18.211.65:41567)
0928 06:40:11.998415 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:11.998417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:11.998486 (+ 69us) server_negotiation.cc:407] Connection header received
0928 06:40:11.998536 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:11.998536 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:11.998546 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:11.998594 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:40:11.998708 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:11.999243 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:11.999979 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:12.000173 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:12.000226 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:12.000344 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:40:12.000375 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:40:12.000417 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:12.000473 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 06:40:12.000491 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 06:40:12.000576 21599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:11.998302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:41567 (local address 127.18.211.65:42699)
0928 06:40:11.998424 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:11.998435 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:11.998492 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:11.998616 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:11.998617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:11.998618 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:40:11.998659 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:11.998661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:11.999323 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:11.999324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:11.999911 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:11.999913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:12.000230 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:12.000231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:12.000278 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:12.000283 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:40:12.000424 (+ 141us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:40:12.000426 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:40:12.000462 (+ 36us) client_negotiation.cc:240] Negotiation successful
0928 06:40:12.000503 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":77,"threads_started":1}
I20260928 06:40:12.002507 21432 raft_consensus.cc:1261] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 20a3dd6652e74d1b8d6890637a957a51 [term 1 FOLLOWER]: Refusing update from remote peer 5a402cacef4e43ca90ed68ac5f334659: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:40:12.002507 21292 raft_consensus.cc:1261] T dbf3b3abf580453fb14e7fa1acb1b6b2 P f030cb407ac5452d9a76231f8449a3a4 [term 1 FOLLOWER]: Refusing update from remote peer 5a402cacef4e43ca90ed68ac5f334659: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:40:12.002707 21562 consensus_queue.cc:1035] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 5a402cacef4e43ca90ed68ac5f334659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:40:12.002830 21520 consensus_queue.cc:1035] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 5a402cacef4e43ca90ed68ac5f334659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:40:12.012758 20995 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:40:12.012866 20995 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:40:12.013433 20995 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b309eba9ace4bb4a324ccd7217e4227: Sending DeleteTablet for 3 replicas of tablet dbf3b3abf580453fb14e7fa1acb1b6b2
I20260928 06:40:12.013744 21412 tablet_service.cc:1514] Processing DeleteTablet for tablet dbf3b3abf580453fb14e7fa1acb1b6b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:12 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:42698
I20260928 06:40:12.013863 21272 tablet_service.cc:1514] Processing DeleteTablet for tablet dbf3b3abf580453fb14e7fa1acb1b6b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:12 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:54890
I20260928 06:40:12.013842 20995 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 06:40:12.013948 21577 tablet_replica.cc:330] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 20a3dd6652e74d1b8d6890637a957a51: stopping tablet replica
W20260928 06:40:12.013979 20995 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:40:12.013993 21577 raft_consensus.cc:2229] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 20a3dd6652e74d1b8d6890637a957a51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:40:12.014027 21577 raft_consensus.cc:2258] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 20a3dd6652e74d1b8d6890637a957a51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:12.014088 21578 tablet_replica.cc:330] T dbf3b3abf580453fb14e7fa1acb1b6b2 P f030cb407ac5452d9a76231f8449a3a4: stopping tablet replica
I20260928 06:40:12.014132 21578 raft_consensus.cc:2229] T dbf3b3abf580453fb14e7fa1acb1b6b2 P f030cb407ac5452d9a76231f8449a3a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:40:12.014161 21578 raft_consensus.cc:2258] T dbf3b3abf580453fb14e7fa1acb1b6b2 P f030cb407ac5452d9a76231f8449a3a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:12.014348 21132 tablet_service.cc:1514] Processing DeleteTablet for tablet dbf3b3abf580453fb14e7fa1acb1b6b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:12 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:49684
I20260928 06:40:12.014431 21576 tablet_replica.cc:330] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 5a402cacef4e43ca90ed68ac5f334659: stopping tablet replica
I20260928 06:40:12.014482 21576 raft_consensus.cc:2229] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 5a402cacef4e43ca90ed68ac5f334659 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:40:12.014488 21577 ts_tablet_manager.cc:1905] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 20a3dd6652e74d1b8d6890637a957a51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:12.014564 21576 raft_consensus.cc:2258] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 5a402cacef4e43ca90ed68ac5f334659 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:12.014844 21576 ts_tablet_manager.cc:1905] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 5a402cacef4e43ca90ed68ac5f334659: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:12.016314 21576 ts_tablet_manager.cc:1918] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 5a402cacef4e43ca90ed68ac5f334659: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:40:12.016371 21576 log.cc:1196] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 5a402cacef4e43ca90ed68ac5f334659: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-0/wal/wals/dbf3b3abf580453fb14e7fa1acb1b6b2
I20260928 06:40:12.016510 21577 ts_tablet_manager.cc:1918] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 20a3dd6652e74d1b8d6890637a957a51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:40:12.016563 21577 log.cc:1196] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 20a3dd6652e74d1b8d6890637a957a51: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-2/wal/wals/dbf3b3abf580453fb14e7fa1acb1b6b2
I20260928 06:40:12.016564 21576 ts_tablet_manager.cc:1939] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 5a402cacef4e43ca90ed68ac5f334659: Deleting consensus metadata
I20260928 06:40:12.016785 21577 ts_tablet_manager.cc:1939] T dbf3b3abf580453fb14e7fa1acb1b6b2 P 20a3dd6652e74d1b8d6890637a957a51: Deleting consensus metadata
I20260928 06:40:12.016793 21272 tablet_service.cc:1467] Processing CreateTablet for tablet e0787949602b405288342bacfc71e96a (DEFAULT_TABLE table=test-table [id=48e9b6321fcd46ed9643a7cbc0ef7321]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:12.016903 21272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0787949602b405288342bacfc71e96a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:12.017043 20980 catalog_manager.cc:4841] TS 20a3dd6652e74d1b8d6890637a957a51 (127.18.211.67:34267): tablet dbf3b3abf580453fb14e7fa1acb1b6b2 (table test-table [id=0a68bf3c06c5488d94c29edd10f84882]) successfully deleted
I20260928 06:40:12.018093 21588 tablet_bootstrap.cc:492] T e0787949602b405288342bacfc71e96a P f030cb407ac5452d9a76231f8449a3a4: Bootstrap starting.
I20260928 06:40:12.018133 21412 tablet_service.cc:1467] Processing CreateTablet for tablet e0787949602b405288342bacfc71e96a (DEFAULT_TABLE table=test-table [id=48e9b6321fcd46ed9643a7cbc0ef7321]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:12.018229 21412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0787949602b405288342bacfc71e96a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:12.018397 21588 tablet_bootstrap.cc:654] T e0787949602b405288342bacfc71e96a P f030cb407ac5452d9a76231f8449a3a4: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:12.019259 20980 catalog_manager.cc:4841] TS 5a402cacef4e43ca90ed68ac5f334659 (127.18.211.65:41567): tablet dbf3b3abf580453fb14e7fa1acb1b6b2 (table test-table [id=0a68bf3c06c5488d94c29edd10f84882]) successfully deleted
I20260928 06:40:12.019364 21587 tablet_bootstrap.cc:492] T e0787949602b405288342bacfc71e96a P 20a3dd6652e74d1b8d6890637a957a51: Bootstrap starting.
I20260928 06:40:12.019474 21578 ts_tablet_manager.cc:1905] T dbf3b3abf580453fb14e7fa1acb1b6b2 P f030cb407ac5452d9a76231f8449a3a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:12.019604 21588 tablet_bootstrap.cc:492] T e0787949602b405288342bacfc71e96a P f030cb407ac5452d9a76231f8449a3a4: No bootstrap required, opened a new log
I20260928 06:40:12.019657 21588 ts_tablet_manager.cc:1397] T e0787949602b405288342bacfc71e96a P f030cb407ac5452d9a76231f8449a3a4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:40:12.019667 21587 tablet_bootstrap.cc:654] T e0787949602b405288342bacfc71e96a P 20a3dd6652e74d1b8d6890637a957a51: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:12.019750 21588 raft_consensus.cc:348] T e0787949602b405288342bacfc71e96a P f030cb407ac5452d9a76231f8449a3a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:12.019804 21588 raft_consensus.cc:374] T e0787949602b405288342bacfc71e96a P f030cb407ac5452d9a76231f8449a3a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:12.019820 21588 raft_consensus.cc:729] T e0787949602b405288342bacfc71e96a P f030cb407ac5452d9a76231f8449a3a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f030cb407ac5452d9a76231f8449a3a4, State: Initialized, Role: FOLLOWER
I20260928 06:40:12.019855 21588 consensus_queue.cc:260] T e0787949602b405288342bacfc71e96a P f030cb407ac5452d9a76231f8449a3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:12.019932 21588 ts_tablet_manager.cc:1428] T e0787949602b405288342bacfc71e96a P f030cb407ac5452d9a76231f8449a3a4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:12.020097 21132 tablet_service.cc:1467] Processing CreateTablet for tablet e0787949602b405288342bacfc71e96a (DEFAULT_TABLE table=test-table [id=48e9b6321fcd46ed9643a7cbc0ef7321]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:12.020181 21132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0787949602b405288342bacfc71e96a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:12.020802 21578 ts_tablet_manager.cc:1918] T dbf3b3abf580453fb14e7fa1acb1b6b2 P f030cb407ac5452d9a76231f8449a3a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:40:12.020855 21578 log.cc:1196] T dbf3b3abf580453fb14e7fa1acb1b6b2 P f030cb407ac5452d9a76231f8449a3a4: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-1/wal/wals/dbf3b3abf580453fb14e7fa1acb1b6b2
I20260928 06:40:12.021036 21578 ts_tablet_manager.cc:1939] T dbf3b3abf580453fb14e7fa1acb1b6b2 P f030cb407ac5452d9a76231f8449a3a4: Deleting consensus metadata
I20260928 06:40:12.021292 21589 tablet_bootstrap.cc:492] T e0787949602b405288342bacfc71e96a P 5a402cacef4e43ca90ed68ac5f334659: Bootstrap starting.
I20260928 06:40:12.021332 20980 catalog_manager.cc:4841] TS f030cb407ac5452d9a76231f8449a3a4 (127.18.211.66:46747): tablet dbf3b3abf580453fb14e7fa1acb1b6b2 (table test-table [id=0a68bf3c06c5488d94c29edd10f84882]) successfully deleted
I20260928 06:40:12.021616 21589 tablet_bootstrap.cc:654] T e0787949602b405288342bacfc71e96a P 5a402cacef4e43ca90ed68ac5f334659: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:12.022112 21589 tablet_bootstrap.cc:492] T e0787949602b405288342bacfc71e96a P 5a402cacef4e43ca90ed68ac5f334659: No bootstrap required, opened a new log
I20260928 06:40:12.022140 21587 tablet_bootstrap.cc:492] T e0787949602b405288342bacfc71e96a P 20a3dd6652e74d1b8d6890637a957a51: No bootstrap required, opened a new log
I20260928 06:40:12.022156 21589 ts_tablet_manager.cc:1397] T e0787949602b405288342bacfc71e96a P 5a402cacef4e43ca90ed68ac5f334659: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:12.022181 21587 ts_tablet_manager.cc:1397] T e0787949602b405288342bacfc71e96a P 20a3dd6652e74d1b8d6890637a957a51: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 06:40:12.022262 21589 raft_consensus.cc:348] T e0787949602b405288342bacfc71e96a P 5a402cacef4e43ca90ed68ac5f334659 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:12.022329 21589 raft_consensus.cc:374] T e0787949602b405288342bacfc71e96a P 5a402cacef4e43ca90ed68ac5f334659 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:12.022351 21589 raft_consensus.cc:729] T e0787949602b405288342bacfc71e96a P 5a402cacef4e43ca90ed68ac5f334659 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a402cacef4e43ca90ed68ac5f334659, State: Initialized, Role: FOLLOWER
I20260928 06:40:12.022315 21587 raft_consensus.cc:348] T e0787949602b405288342bacfc71e96a P 20a3dd6652e74d1b8d6890637a957a51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:12.022374 21587 raft_consensus.cc:374] T e0787949602b405288342bacfc71e96a P 20a3dd6652e74d1b8d6890637a957a51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:12.022393 21587 raft_consensus.cc:729] T e0787949602b405288342bacfc71e96a P 20a3dd6652e74d1b8d6890637a957a51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a3dd6652e74d1b8d6890637a957a51, State: Initialized, Role: FOLLOWER
I20260928 06:40:12.022400 21589 consensus_queue.cc:260] T e0787949602b405288342bacfc71e96a P 5a402cacef4e43ca90ed68ac5f334659 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:12.022435 21587 consensus_queue.cc:260] T e0787949602b405288342bacfc71e96a P 20a3dd6652e74d1b8d6890637a957a51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:12.022486 21589 ts_tablet_manager.cc:1428] T e0787949602b405288342bacfc71e96a P 5a402cacef4e43ca90ed68ac5f334659: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:12.022519 21587 ts_tablet_manager.cc:1428] T e0787949602b405288342bacfc71e96a P 20a3dd6652e74d1b8d6890637a957a51: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:12.058656 21562 raft_consensus.cc:482] T e0787949602b405288342bacfc71e96a P 5a402cacef4e43ca90ed68ac5f334659 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:40:12.058758 21562 raft_consensus.cc:504] T e0787949602b405288342bacfc71e96a P 5a402cacef4e43ca90ed68ac5f334659 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:12.058911 21562 leader_election.cc:288] T e0787949602b405288342bacfc71e96a P 5a402cacef4e43ca90ed68ac5f334659 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f030cb407ac5452d9a76231f8449a3a4 (127.18.211.66:46747), 20a3dd6652e74d1b8d6890637a957a51 (127.18.211.67:34267)
I20260928 06:40:12.059130 21292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0787949602b405288342bacfc71e96a" candidate_uuid: "5a402cacef4e43ca90ed68ac5f334659" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f030cb407ac5452d9a76231f8449a3a4" is_pre_election: true
I20260928 06:40:12.059134 21432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0787949602b405288342bacfc71e96a" candidate_uuid: "5a402cacef4e43ca90ed68ac5f334659" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a3dd6652e74d1b8d6890637a957a51" is_pre_election: true
I20260928 06:40:12.059229 21292 raft_consensus.cc:2454] T e0787949602b405288342bacfc71e96a P f030cb407ac5452d9a76231f8449a3a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a402cacef4e43ca90ed68ac5f334659 in term 0.
I20260928 06:40:12.059229 21432 raft_consensus.cc:2454] T e0787949602b405288342bacfc71e96a P 20a3dd6652e74d1b8d6890637a957a51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a402cacef4e43ca90ed68ac5f334659 in term 0.
I20260928 06:40:12.059404 21084 leader_election.cc:302] T e0787949602b405288342bacfc71e96a P 5a402cacef4e43ca90ed68ac5f334659 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a3dd6652e74d1b8d6890637a957a51, 5a402cacef4e43ca90ed68ac5f334659; no voters:
I20260928 06:40:12.059530 21562 raft_consensus.cc:2781] T e0787949602b405288342bacfc71e96a P 5a402cacef4e43ca90ed68ac5f334659 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:40:12.059579 21562 raft_consensus.cc:482] T e0787949602b405288342bacfc71e96a P 5a402cacef4e43ca90ed68ac5f334659 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:40:12.059599 21562 raft_consensus.cc:3037] T e0787949602b405288342bacfc71e96a P 5a402cacef4e43ca90ed68ac5f334659 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:12.060231 21562 raft_consensus.cc:504] T e0787949602b405288342bacfc71e96a P 5a402cacef4e43ca90ed68ac5f334659 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:12.060376 21562 leader_election.cc:288] T e0787949602b405288342bacfc71e96a P 5a402cacef4e43ca90ed68ac5f334659 [CANDIDATE]: Term 1 election: Requested vote from peers f030cb407ac5452d9a76231f8449a3a4 (127.18.211.66:46747), 20a3dd6652e74d1b8d6890637a957a51 (127.18.211.67:34267)
I20260928 06:40:12.060561 21292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0787949602b405288342bacfc71e96a" candidate_uuid: "5a402cacef4e43ca90ed68ac5f334659" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f030cb407ac5452d9a76231f8449a3a4"
I20260928 06:40:12.060642 21292 raft_consensus.cc:3037] T e0787949602b405288342bacfc71e96a P f030cb407ac5452d9a76231f8449a3a4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:12.060621 21432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0787949602b405288342bacfc71e96a" candidate_uuid: "5a402cacef4e43ca90ed68ac5f334659" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a3dd6652e74d1b8d6890637a957a51"
I20260928 06:40:12.060714 21432 raft_consensus.cc:3037] T e0787949602b405288342bacfc71e96a P 20a3dd6652e74d1b8d6890637a957a51 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:12.061250 21292 raft_consensus.cc:2454] T e0787949602b405288342bacfc71e96a P f030cb407ac5452d9a76231f8449a3a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a402cacef4e43ca90ed68ac5f334659 in term 1.
I20260928 06:40:12.061416 21084 leader_election.cc:302] T e0787949602b405288342bacfc71e96a P 5a402cacef4e43ca90ed68ac5f334659 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a402cacef4e43ca90ed68ac5f334659, f030cb407ac5452d9a76231f8449a3a4; no voters:
I20260928 06:40:12.061473 21432 raft_consensus.cc:2454] T e0787949602b405288342bacfc71e96a P 20a3dd6652e74d1b8d6890637a957a51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a402cacef4e43ca90ed68ac5f334659 in term 1.
I20260928 06:40:12.061517 21562 raft_consensus.cc:2781] T e0787949602b405288342bacfc71e96a P 5a402cacef4e43ca90ed68ac5f334659 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:12.061559 21562 raft_consensus.cc:686] T e0787949602b405288342bacfc71e96a P 5a402cacef4e43ca90ed68ac5f334659 [term 1 LEADER]: Becoming Leader. State: Replica: 5a402cacef4e43ca90ed68ac5f334659, State: Running, Role: LEADER
I20260928 06:40:12.061606 21562 consensus_queue.cc:237] T e0787949602b405288342bacfc71e96a P 5a402cacef4e43ca90ed68ac5f334659 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } }
I20260928 06:40:12.062173 20994 catalog_manager.cc:5495] T e0787949602b405288342bacfc71e96a P 5a402cacef4e43ca90ed68ac5f334659 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a402cacef4e43ca90ed68ac5f334659 (127.18.211.65). New cstate: current_term: 1 leader_uuid: "5a402cacef4e43ca90ed68ac5f334659" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a402cacef4e43ca90ed68ac5f334659" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 } health_report { overall_health: UNKNOWN } } }
I20260928 06:40:12.079404 21432 raft_consensus.cc:1261] T e0787949602b405288342bacfc71e96a P 20a3dd6652e74d1b8d6890637a957a51 [term 1 FOLLOWER]: Refusing update from remote peer 5a402cacef4e43ca90ed68ac5f334659: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:40:12.079437 21292 raft_consensus.cc:1261] T e0787949602b405288342bacfc71e96a P f030cb407ac5452d9a76231f8449a3a4 [term 1 FOLLOWER]: Refusing update from remote peer 5a402cacef4e43ca90ed68ac5f334659: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:40:12.079608 21557 consensus_queue.cc:1035] T e0787949602b405288342bacfc71e96a P 5a402cacef4e43ca90ed68ac5f334659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a3dd6652e74d1b8d6890637a957a51" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:40:12.079741 21562 consensus_queue.cc:1035] T e0787949602b405288342bacfc71e96a P 5a402cacef4e43ca90ed68ac5f334659 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f030cb407ac5452d9a76231f8449a3a4" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 06:40:12.083981 21555 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 06:40:13.600339 20994 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:40:13.600456 20994 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:40:13.601316 20994 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b309eba9ace4bb4a324ccd7217e4227: Sending DeleteTablet for 3 replicas of tablet e0787949602b405288342bacfc71e96a
I20260928 06:40:13.601599 21272 tablet_service.cc:1514] Processing DeleteTablet for tablet e0787949602b405288342bacfc71e96a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:13 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:54890
I20260928 06:40:13.601652 21132 tablet_service.cc:1514] Processing DeleteTablet for tablet e0787949602b405288342bacfc71e96a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:13 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:49684
I20260928 06:40:13.601717 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 21067
I20260928 06:40:13.601799 21412 tablet_service.cc:1514] Processing DeleteTablet for tablet e0787949602b405288342bacfc71e96a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:13 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:42698
I20260928 06:40:13.602540 21630 tablet_replica.cc:330] T e0787949602b405288342bacfc71e96a P 20a3dd6652e74d1b8d6890637a957a51: stopping tablet replica
I20260928 06:40:13.602593 21630 raft_consensus.cc:2229] T e0787949602b405288342bacfc71e96a P 20a3dd6652e74d1b8d6890637a957a51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:40:13.602634 21630 raft_consensus.cc:2258] T e0787949602b405288342bacfc71e96a P 20a3dd6652e74d1b8d6890637a957a51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:13.602752 21628 tablet_replica.cc:330] T e0787949602b405288342bacfc71e96a P f030cb407ac5452d9a76231f8449a3a4: stopping tablet replica
I20260928 06:40:13.602808 21628 raft_consensus.cc:2229] T e0787949602b405288342bacfc71e96a P f030cb407ac5452d9a76231f8449a3a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:40:13.602869 21628 raft_consensus.cc:2258] T e0787949602b405288342bacfc71e96a P f030cb407ac5452d9a76231f8449a3a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:13.602881 21630 ts_tablet_manager.cc:1905] T e0787949602b405288342bacfc71e96a P 20a3dd6652e74d1b8d6890637a957a51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:13.603277 21628 ts_tablet_manager.cc:1905] T e0787949602b405288342bacfc71e96a P f030cb407ac5452d9a76231f8449a3a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:13.604498 21628 ts_tablet_manager.cc:1918] T e0787949602b405288342bacfc71e96a P f030cb407ac5452d9a76231f8449a3a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:40:13.604549 21628 log.cc:1196] T e0787949602b405288342bacfc71e96a P f030cb407ac5452d9a76231f8449a3a4: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-1/wal/wals/e0787949602b405288342bacfc71e96a
I20260928 06:40:13.604604 21630 ts_tablet_manager.cc:1918] T e0787949602b405288342bacfc71e96a P 20a3dd6652e74d1b8d6890637a957a51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 06:40:13.604671 21630 log.cc:1196] T e0787949602b405288342bacfc71e96a P 20a3dd6652e74d1b8d6890637a957a51: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790577602522428-19277-0/minicluster-data/ts-2/wal/wals/e0787949602b405288342bacfc71e96a
I20260928 06:40:13.604794 21628 ts_tablet_manager.cc:1939] T e0787949602b405288342bacfc71e96a P f030cb407ac5452d9a76231f8449a3a4: Deleting consensus metadata
I20260928 06:40:13.604943 21630 ts_tablet_manager.cc:1939] T e0787949602b405288342bacfc71e96a P 20a3dd6652e74d1b8d6890637a957a51: Deleting consensus metadata
I20260928 06:40:13.605082 20980 catalog_manager.cc:4841] TS f030cb407ac5452d9a76231f8449a3a4 (127.18.211.66:46747): tablet e0787949602b405288342bacfc71e96a (table test-table [id=48e9b6321fcd46ed9643a7cbc0ef7321]) successfully deleted
I20260928 06:40:13.605264 20980 catalog_manager.cc:4841] TS 20a3dd6652e74d1b8d6890637a957a51 (127.18.211.67:34267): tablet e0787949602b405288342bacfc71e96a (table test-table [id=48e9b6321fcd46ed9643a7cbc0ef7321]) successfully deleted
W20260928 06:40:13.609104 20980 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.211.65:41567 (error 108)
I20260928 06:40:13.609297 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 21207
I20260928 06:40:13.610049 21632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:13.609439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:41567 (local address 127.0.0.1:49720)
0928 06:40:13.609969 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:13.610012 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.65:41567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":56,"threads_started":1}
W20260928 06:40:13.610203 20980 catalog_manager.cc:4584] TS 5a402cacef4e43ca90ed68ac5f334659 (127.18.211.65:41567): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e0787949602b405288342bacfc71e96a: Network error: Client connection negotiation failed: client connection to 127.18.211.65:41567: connect: Connection refused (error 111)
I20260928 06:40:13.615332 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 21347
I20260928 06:40:13.619467 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 20963
2026-09-28T06:40:13Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5528 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:40:13 dist-test-slave-b28r krb5kdc[21635](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:40:13 dist-test-slave-b28r krb5kdc[21635](info): set up 2 sockets
Sep 28 06:40:13 dist-test-slave-b28r krb5kdc[21635](info): commencing operation
krb5kdc: starting...
W20260928 06:40:14.563727 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:40:14 dist-test-slave-b28r krb5kdc[21635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:40:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:40:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.211.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
I20260928 06:40:14.677541 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:40367
--webserver_interface=127.18.211.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:36593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:40367
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:14.755050 21648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:14.755264 21648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:14.755285 21648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:14.756376 21648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:40:14.756407 21648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:14.756419 21648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:14.756431 21648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:40:14.756441 21648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:40:14.757833 21648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:36593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:40367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:40367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:14.758023 21648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:14.758185 21648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:14.762828 21654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:14.762837 21656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:14.762861 21653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:14.763025 21648 server_base.cc:1034] running on GCE node
I20260928 06:40:14.763432 21648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:14.763729 21648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:14.764895 21648 hybrid_clock.cc:648] HybridClock initialized: now 1790577614764879 us; error 30 us; skew 500 ppm
Sep 28 06:40:14 dist-test-slave-b28r krb5kdc[21635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577614, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:14.771629 21648 init.cc:377] Logged in from keytab as kudu/127.18.211.126@KRBTEST.COM (short username kudu)
I20260928 06:40:14.772055 21648 webserver.cc:460] Webserver started at http://127.18.211.126:44001/ using document root <none> and password file <none>
I20260928 06:40:14.772315 21648 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:14.772367 21648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:14.772485 21648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:14.773535 21648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "e3b09f89747444fbb8bd0714ea4287e0"
format_stamp: "Formatted at 2026-09-28 06:40:14 on dist-test-slave-b28r"
server_key: "c17d33df20d1158f1f4a0ba46ff7fc59"
server_key_iv: "76b847e8e928e7ae5533d52406d9e6f3"
server_key_version: "encryptionkey@0"
I20260928 06:40:14.773869 21648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "e3b09f89747444fbb8bd0714ea4287e0"
format_stamp: "Formatted at 2026-09-28 06:40:14 on dist-test-slave-b28r"
server_key: "c17d33df20d1158f1f4a0ba46ff7fc59"
server_key_iv: "76b847e8e928e7ae5533d52406d9e6f3"
server_key_version: "encryptionkey@0"
I20260928 06:40:14.775148 21648 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 06:40:14.775995 21663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:14.776201 21648 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:14.776288 21648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "e3b09f89747444fbb8bd0714ea4287e0"
format_stamp: "Formatted at 2026-09-28 06:40:14 on dist-test-slave-b28r"
server_key: "c17d33df20d1158f1f4a0ba46ff7fc59"
server_key_iv: "76b847e8e928e7ae5533d52406d9e6f3"
server_key_version: "encryptionkey@0"
I20260928 06:40:14.776350 21648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:14.786669 21648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:14.788672 21648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:14.788808 21648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:14.792546 21648 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:40367
I20260928 06:40:14.792589 21715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:40367 every 8 connection(s)
I20260928 06:40:14.792904 21648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:40:14.793768 21716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:14.796015 21716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3b09f89747444fbb8bd0714ea4287e0: Bootstrap starting.
I20260928 06:40:14.796547 21716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3b09f89747444fbb8bd0714ea4287e0: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:14.796815 21716 log.cc:824] T 00000000000000000000000000000000 P e3b09f89747444fbb8bd0714ea4287e0: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:14.797582 21716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3b09f89747444fbb8bd0714ea4287e0: No bootstrap required, opened a new log
I20260928 06:40:14.799255 21716 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3b09f89747444fbb8bd0714ea4287e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3b09f89747444fbb8bd0714ea4287e0" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 40367 } }
I20260928 06:40:14.799393 21716 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3b09f89747444fbb8bd0714ea4287e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:14.799414 21716 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3b09f89747444fbb8bd0714ea4287e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3b09f89747444fbb8bd0714ea4287e0, State: Initialized, Role: FOLLOWER
I20260928 06:40:14.799558 21716 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3b09f89747444fbb8bd0714ea4287e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3b09f89747444fbb8bd0714ea4287e0" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 40367 } }
I20260928 06:40:14.799640 21716 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3b09f89747444fbb8bd0714ea4287e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:40:14.799677 21716 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3b09f89747444fbb8bd0714ea4287e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:40:14.799732 21716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3b09f89747444fbb8bd0714ea4287e0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:14.800225 21716 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3b09f89747444fbb8bd0714ea4287e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3b09f89747444fbb8bd0714ea4287e0" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 40367 } }
I20260928 06:40:14.800319 21716 leader_election.cc:302] T 00000000000000000000000000000000 P e3b09f89747444fbb8bd0714ea4287e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3b09f89747444fbb8bd0714ea4287e0; no voters:
I20260928 06:40:14.800473 21716 leader_election.cc:288] T 00000000000000000000000000000000 P e3b09f89747444fbb8bd0714ea4287e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:40:14.800510 21719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3b09f89747444fbb8bd0714ea4287e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:14.800666 21719 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3b09f89747444fbb8bd0714ea4287e0 [term 1 LEADER]: Becoming Leader. State: Replica: e3b09f89747444fbb8bd0714ea4287e0, State: Running, Role: LEADER
I20260928 06:40:14.800731 21716 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3b09f89747444fbb8bd0714ea4287e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:40:14.800755 21719 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3b09f89747444fbb8bd0714ea4287e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3b09f89747444fbb8bd0714ea4287e0" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 40367 } }
I20260928 06:40:14.801137 21721 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3b09f89747444fbb8bd0714ea4287e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3b09f89747444fbb8bd0714ea4287e0. Latest consensus state: current_term: 1 leader_uuid: "e3b09f89747444fbb8bd0714ea4287e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3b09f89747444fbb8bd0714ea4287e0" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 40367 } } }
I20260928 06:40:14.801280 21720 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3b09f89747444fbb8bd0714ea4287e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3b09f89747444fbb8bd0714ea4287e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3b09f89747444fbb8bd0714ea4287e0" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 40367 } } }
I20260928 06:40:14.801345 21720 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3b09f89747444fbb8bd0714ea4287e0 [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:14.801275 21721 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3b09f89747444fbb8bd0714ea4287e0 [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:14.801733 21728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:40:14.801930 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 21648
I20260928 06:40:14.802016 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:40:14.802177 19277 external_mini_cluster.cc:1428] Setting key eb5719f50afb3fa53560218e45ddd673
I20260928 06:40:14.802244 21728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:40:14.803885 21728 catalog_manager.cc:1347] Generated new cluster ID: f2eb036cb23540bba9f6cd553c227d8b
I20260928 06:40:14.803934 21728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:40:14 dist-test-slave-b28r krb5kdc[21635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:14.813727 21738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:14.802624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44802 (local address 127.18.211.126:40367)
0928 06:40:14.803203 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:14.803212 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:14.803239 (+ 27us) server_negotiation.cc:407] Connection header received
0928 06:40:14.803356 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:14.803363 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:14.803418 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:14.803555 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:14.804305 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:14.805680 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:14.806732 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:14.807354 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:14.807447 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:14.810185 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:14.810190 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:14.810190 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:14.810204 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:14.812139 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:14.812503 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:14.812506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:14.812506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:14.812623 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:14.812789 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:14.812790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:14.812790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:14.813094 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:14.813269 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:14.813340 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 06:40:14.813479 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":79,"threads_started":1}
I20260928 06:40:14.817569 21728 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:40:14.818207 21728 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:40:14.823369 21728 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3b09f89747444fbb8bd0714ea4287e0: Generated new TSK 0
I20260928 06:40:14.823541 21728 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.211.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab.
Entry for principal kudu/127.18.211.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab.
I20260928 06:40:14.851665 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.65:0
--local_ip_for_outbound_sockets=127.18.211.65
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:40367
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:36593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:14.930631 21742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:14.930827 21742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:14.930850 21742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:14.931986 21742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:14.932026 21742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:14.932051 21742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.65
I20260928 06:40:14.933409 21742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:36593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:40367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.65
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:14.933593 21742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:14.933769 21742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:14.938453 21748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:14.938445 21747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:14.938445 21750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:14.938668 21742 server_base.cc:1034] running on GCE node
I20260928 06:40:14.938889 21742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:14.939126 21742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:14.940317 21742 hybrid_clock.cc:648] HybridClock initialized: now 1790577614940292 us; error 33 us; skew 500 ppm
Sep 28 06:40:14 dist-test-slave-b28r krb5kdc[21635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577614, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:14.942633 21742 init.cc:377] Logged in from keytab as kudu/127.18.211.65@KRBTEST.COM (short username kudu)
I20260928 06:40:14.943037 21742 webserver.cc:460] Webserver started at http://127.18.211.65:46321/ using document root <none> and password file <none>
I20260928 06:40:14.943269 21742 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:14.943315 21742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:14.943434 21742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:14.944408 21742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-0/data/instance:
uuid: "6d2d958abbae45d88295018fcb3d6ca3"
format_stamp: "Formatted at 2026-09-28 06:40:14 on dist-test-slave-b28r"
server_key: "adf31ebb598eb91a37de5d88096402ef"
server_key_iv: "c4878eaefbac53d378ec81ddcbac623d"
server_key_version: "encryptionkey@0"
I20260928 06:40:14.944725 21742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance:
uuid: "6d2d958abbae45d88295018fcb3d6ca3"
format_stamp: "Formatted at 2026-09-28 06:40:14 on dist-test-slave-b28r"
server_key: "adf31ebb598eb91a37de5d88096402ef"
server_key_iv: "c4878eaefbac53d378ec81ddcbac623d"
server_key_version: "encryptionkey@0"
I20260928 06:40:14.945995 21742 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:40:14.946828 21757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:14.946995 21742 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:14.947109 21742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-0/wal
uuid: "6d2d958abbae45d88295018fcb3d6ca3"
format_stamp: "Formatted at 2026-09-28 06:40:14 on dist-test-slave-b28r"
server_key: "adf31ebb598eb91a37de5d88096402ef"
server_key_iv: "c4878eaefbac53d378ec81ddcbac623d"
server_key_version: "encryptionkey@0"
I20260928 06:40:14.947189 21742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:14.961601 21742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:14.963537 21742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:14.963673 21742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:14.963896 21742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:14.964195 21742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:14.964227 21742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:14.964257 21742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:14.964279 21742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:14.969830 21742 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.65:45559
I20260928 06:40:14.969890 21870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.65:45559 every 8 connection(s)
I20260928 06:40:14.970217 21742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
I20260928 06:40:14.976150 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 21742
I20260928 06:40:14.976243 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance
Sep 28 06:40:14 dist-test-slave-b28r krb5kdc[21635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577614, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:14.976428 19277 external_mini_cluster.cc:1428] Setting key 87d9349173a493301df477a2234e28c5
I20260928 06:40:14.979749 21738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:14.970883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:47723 (local address 127.18.211.126:40367)
0928 06:40:14.970940 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:14.970941 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:14.971213 (+ 272us) server_negotiation.cc:407] Connection header received
0928 06:40:14.971715 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:14.971716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:14.971725 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:14.971769 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:14.972779 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:14.973284 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:14.974331 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:14.974547 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:14.974607 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:14.976830 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:14.976831 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:14.976832 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:14.976840 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:14.978220 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:14.978701 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:14.978702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:14.978702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:14.978763 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:14.979021 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:14.979022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:14.979022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:14.979168 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:14.979263 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:14.979595 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 06:40:14.979681 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 06:40:14.979930 21873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:14.970856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:40367 (local address 127.18.211.65:47723)
0928 06:40:14.971036 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:14.971111 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:14.971615 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:14.971841 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:14.971845 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:14.972499 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:14.972700 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:14.972705 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:14.973402 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:14.973403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:14.974231 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:14.974233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:14.974645 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:14.974646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:14.974768 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:14.975301 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:14.975309 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:14.976739 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:14.978308 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:14.978315 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:14.978316 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:14.978617 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:14.978818 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:14.978818 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:14.978819 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:14.978958 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:14.979256 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:14.979263 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:14.979421 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 06:40:14.979543 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 06:40:14.979705 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":82,"threads_started":1}
I20260928 06:40:14.980521 21871 heartbeater.cc:347] Connected to a master server at 127.18.211.126:40367
I20260928 06:40:14.980620 21871 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:14.980810 21871 heartbeater.cc:511] Master 127.18.211.126:40367 requested a full tablet report, sending...
I20260928 06:40:14.981330 21680 ts_manager.cc:194] Registered new tserver with Master: 6d2d958abbae45d88295018fcb3d6ca3 (127.18.211.65:45559)
I20260928 06:40:14.981901 21680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.65@KRBTEST.COM'} at 127.18.211.65:47723
WARNING: no policy specified for kudu/127.18.211.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab.
Entry for principal kudu/127.18.211.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab.
I20260928 06:40:14.998994 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.66:0
--local_ip_for_outbound_sockets=127.18.211.66
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:40367
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:36593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:15.077093 21876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:15.077349 21876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:15.077373 21876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:15.078543 21876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:15.078585 21876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:15.078610 21876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.66
I20260928 06:40:15.080132 21876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:36593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:40367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.66
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:15.080336 21876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:15.080550 21876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:15.085357 21884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:15.085481 21881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:15.085357 21882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:15.085538 21876 server_base.cc:1034] running on GCE node
I20260928 06:40:15.085750 21876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:15.085985 21876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:15.087157 21876 hybrid_clock.cc:648] HybridClock initialized: now 1790577615087116 us; error 52 us; skew 500 ppm
Sep 28 06:40:15 dist-test-slave-b28r krb5kdc[21635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577615, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:15.089546 21876 init.cc:377] Logged in from keytab as kudu/127.18.211.66@KRBTEST.COM (short username kudu)
I20260928 06:40:15.089967 21876 webserver.cc:460] Webserver started at http://127.18.211.66:42143/ using document root <none> and password file <none>
I20260928 06:40:15.090174 21876 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:15.090234 21876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:15.090364 21876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:15.091420 21876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-1/data/instance:
uuid: "ad3114d3d9664363832236ea2978dfe6"
format_stamp: "Formatted at 2026-09-28 06:40:15 on dist-test-slave-b28r"
server_key: "f00f8c4c865cab552372e52b6d8d8aa4"
server_key_iv: "5a15e957b5306b61079cfc7dc5b7a080"
server_key_version: "encryptionkey@0"
I20260928 06:40:15.091753 21876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance:
uuid: "ad3114d3d9664363832236ea2978dfe6"
format_stamp: "Formatted at 2026-09-28 06:40:15 on dist-test-slave-b28r"
server_key: "f00f8c4c865cab552372e52b6d8d8aa4"
server_key_iv: "5a15e957b5306b61079cfc7dc5b7a080"
server_key_version: "encryptionkey@0"
I20260928 06:40:15.092988 21876 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:40:15.093832 21891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:15.094036 21876 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:15.094115 21876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-1/wal
uuid: "ad3114d3d9664363832236ea2978dfe6"
format_stamp: "Formatted at 2026-09-28 06:40:15 on dist-test-slave-b28r"
server_key: "f00f8c4c865cab552372e52b6d8d8aa4"
server_key_iv: "5a15e957b5306b61079cfc7dc5b7a080"
server_key_version: "encryptionkey@0"
I20260928 06:40:15.094184 21876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:15.123118 21876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:15.125368 21876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:15.125546 21876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:15.125808 21876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:15.126157 21876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:15.126197 21876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:15.126231 21876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:15.126255 21876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:15.132341 21876 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.66:37217
I20260928 06:40:15.132417 22004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.66:37217 every 8 connection(s)
I20260928 06:40:15.132722 21876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
I20260928 06:40:15.133574 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 21876
I20260928 06:40:15.133674 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance
I20260928 06:40:15.133909 19277 external_mini_cluster.cc:1428] Setting key da25a666ac76817f0958cf0147a7a08e
Sep 28 06:40:15 dist-test-slave-b28r krb5kdc[21635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577615, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.211.67@KRBTEST.COM; defaulting to no policy
I20260928 06:40:15.142840 21738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:15.133476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:48911 (local address 127.18.211.126:40367)
0928 06:40:15.133736 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:15.133738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:15.133760 (+ 22us) server_negotiation.cc:407] Connection header received
0928 06:40:15.134378 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:15.134379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:15.134390 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:15.134455 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:15.135708 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:15.136209 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:15.137473 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:15.137654 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:15.137732 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:15.140074 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:15.140075 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:15.140075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:15.140083 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:15.141433 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:15.141886 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:15.141887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:15.141887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:15.141941 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:15.142204 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:15.142204 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:15.142205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:15.142343 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:15.142395 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:15.142683 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 06:40:15.142763 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228}
I20260928 06:40:15.142925 22007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:15.133428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:40367 (local address 127.18.211.66:48911)
0928 06:40:15.133598 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:15.133652 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:15.134280 (+ 628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:15.134513 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:15.134517 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:15.135328 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:15.135610 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:15.135617 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:15.136290 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:15.136291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:15.137401 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:15.137404 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:15.137747 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:15.137748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:15.137877 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:15.138359 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:15.138367 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:15.139930 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:15.141513 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:15.141521 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:15.141523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:15.141802 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:15.141998 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:15.141999 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:15.141999 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:15.142134 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:15.142407 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:15.142412 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:15.142531 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 06:40:15.142619 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 06:40:15.142738 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
I20260928 06:40:15.143411 22005 heartbeater.cc:347] Connected to a master server at 127.18.211.126:40367
I20260928 06:40:15.143508 22005 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:15.143683 22005 heartbeater.cc:511] Master 127.18.211.126:40367 requested a full tablet report, sending...
I20260928 06:40:15.144058 21680 ts_manager.cc:194] Registered new tserver with Master: ad3114d3d9664363832236ea2978dfe6 (127.18.211.66:37217)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.67@KRBTEST.COM" created.
I20260928 06:40:15.144510 21680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.66@KRBTEST.COM'} at 127.18.211.66:48911
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab.
Entry for principal kudu/127.18.211.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab.
I20260928 06:40:15.157670 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.67:0
--local_ip_for_outbound_sockets=127.18.211.67
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:40367
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:36593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:15.237582 22010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:15.237800 22010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:15.237833 22010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:15.238992 22010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:15.239034 22010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:15.239069 22010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.67
I20260928 06:40:15.240582 22010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:36593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:40367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.67
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:15.240820 22010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:15.241098 22010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:15.246213 22015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:15.246234 22016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:15.246213 22018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:15.246435 22010 server_base.cc:1034] running on GCE node
I20260928 06:40:15.246654 22010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:15.246920 22010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:15.248059 22010 hybrid_clock.cc:648] HybridClock initialized: now 1790577615248038 us; error 34 us; skew 500 ppm
Sep 28 06:40:15 dist-test-slave-b28r krb5kdc[21635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577615, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:15.250564 22010 init.cc:377] Logged in from keytab as kudu/127.18.211.67@KRBTEST.COM (short username kudu)
I20260928 06:40:15.250985 22010 webserver.cc:460] Webserver started at http://127.18.211.67:46037/ using document root <none> and password file <none>
I20260928 06:40:15.251241 22010 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:15.251293 22010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:15.251415 22010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:15.252460 22010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-2/data/instance:
uuid: "c3ec692b597241909fa5050f2cf3fffd"
format_stamp: "Formatted at 2026-09-28 06:40:15 on dist-test-slave-b28r"
server_key: "e4839cabca69c3056285aace99a42ede"
server_key_iv: "8f95b6ea6513ff2e291e816d5b525e1b"
server_key_version: "encryptionkey@0"
I20260928 06:40:15.252830 22010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance:
uuid: "c3ec692b597241909fa5050f2cf3fffd"
format_stamp: "Formatted at 2026-09-28 06:40:15 on dist-test-slave-b28r"
server_key: "e4839cabca69c3056285aace99a42ede"
server_key_iv: "8f95b6ea6513ff2e291e816d5b525e1b"
server_key_version: "encryptionkey@0"
I20260928 06:40:15.254110 22010 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:40:15.254901 22025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:15.255071 22010 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:15.255195 22010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-2/wal
uuid: "c3ec692b597241909fa5050f2cf3fffd"
format_stamp: "Formatted at 2026-09-28 06:40:15 on dist-test-slave-b28r"
server_key: "e4839cabca69c3056285aace99a42ede"
server_key_iv: "8f95b6ea6513ff2e291e816d5b525e1b"
server_key_version: "encryptionkey@0"
I20260928 06:40:15.255276 22010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:15.275296 22010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:15.277326 22010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:15.277480 22010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:15.277726 22010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:15.278049 22010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:15.278088 22010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:15.278122 22010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:15.278151 22010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:15.284395 22010 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.67:35003
I20260928 06:40:15.284447 22138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.67:35003 every 8 connection(s)
I20260928 06:40:15.284808 22010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:40:15 dist-test-slave-b28r krb5kdc[21635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577615, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:15.292634 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 22010
I20260928 06:40:15.292745 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance
I20260928 06:40:15.292963 19277 external_mini_cluster.cc:1428] Setting key cea9b681e043e92f48af80e4b38e04f4
I20260928 06:40:15.294353 21738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:15.285539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:58921 (local address 127.18.211.126:40367)
0928 06:40:15.285593 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:15.285595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:15.285798 (+ 203us) server_negotiation.cc:407] Connection header received
0928 06:40:15.286317 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:15.286318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:15.286328 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:15.286392 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:15.287441 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:15.287954 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:15.288964 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:15.289138 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:15.289181 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:15.291429 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:15.291430 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:15.291431 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:15.291439 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:15.292762 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:15.293201 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:15.293203 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:15.293203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:15.293276 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:15.293565 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:15.293567 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:15.293567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:15.293707 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:15.293774 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:15.294124 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 06:40:15.294211 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 06:40:15.294509 22141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:15.285486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:40367 (local address 127.18.211.67:58921)
0928 06:40:15.285664 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:15.285708 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:15.286226 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:15.286458 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:15.286461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:15.287031 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:15.287328 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:15.287334 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:15.288048 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:15.288049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:15.288899 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:15.288902 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:15.289238 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:15.289240 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:15.289363 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:15.289822 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:15.289830 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:15.291335 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:15.292857 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:15.292862 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:15.292864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:15.293116 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:15.293338 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:15.293339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:15.293340 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:15.293493 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:15.293774 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:15.293781 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:15.293945 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 06:40:15.294090 (+ 145us) client_negotiation.cc:240] Negotiation successful
0928 06:40:15.294285 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":74,"threads_started":1}
I20260928 06:40:15.294988 22139 heartbeater.cc:347] Connected to a master server at 127.18.211.126:40367
I20260928 06:40:15.295069 22139 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:15.295266 22139 heartbeater.cc:511] Master 127.18.211.126:40367 requested a full tablet report, sending...
I20260928 06:40:15.295621 21680 ts_manager.cc:194] Registered new tserver with Master: c3ec692b597241909fa5050f2cf3fffd (127.18.211.67:35003)
I20260928 06:40:15.295971 21680 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.67@KRBTEST.COM'} at 127.18.211.67:58921
I20260928 06:40:15.304733 19277 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 06:40:15.315338 21738 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:15.314415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44806 (local address 127.18.211.126:40367)
0928 06:40:15.314475 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:15.314476 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:15.314559 (+ 83us) server_negotiation.cc:407] Connection header received
0928 06:40:15.314629 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:15.314630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:15.314640 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:15.314684 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:15.315222 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44806: BlockingRecv error: recv got EOF from 127.0.0.1:44806 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 06:40:15.316092 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 21742
I20260928 06:40:15.320344 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 21876
I20260928 06:40:15.323997 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 22010
I20260928 06:40:15.327702 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 21648
2026-09-28T06:40:15Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1707 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:40:15 dist-test-slave-b28r krb5kdc[22151](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:40:15 dist-test-slave-b28r krb5kdc[22151](info): set up 2 sockets
Sep 28 06:40:15 dist-test-slave-b28r krb5kdc[22151](info): commencing operation
krb5kdc: starting...
W20260928 06:40:16.271798 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:40:16 dist-test-slave-b28r krb5kdc[22151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:40:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:40:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.211.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
I20260928 06:40:16.379695 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:42181
--webserver_interface=127.18.211.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:41563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:42181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:16.458894 22164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:16.459120 22164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:16.459152 22164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:16.460279 22164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:40:16.460316 22164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:16.460335 22164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:16.460350 22164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:40:16.460364 22164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:40:16.461808 22164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:41563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:42181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:42181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:16.462041 22164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:16.462247 22164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:16.466818 22169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:16.466830 22170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:16.466884 22164 server_base.cc:1034] running on GCE node
W20260928 06:40:16.466835 22172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:16.467291 22164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:16.467563 22164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:16.468703 22164 hybrid_clock.cc:648] HybridClock initialized: now 1790577616468685 us; error 38 us; skew 500 ppm
Sep 28 06:40:16 dist-test-slave-b28r krb5kdc[22151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577616, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:16.471115 22164 init.cc:377] Logged in from keytab as kudu/127.18.211.126@KRBTEST.COM (short username kudu)
I20260928 06:40:16.471544 22164 webserver.cc:460] Webserver started at http://127.18.211.126:34213/ using document root <none> and password file <none>
I20260928 06:40:16.471732 22164 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:16.471776 22164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:16.471866 22164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:16.472851 22164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "331ba69165b443a9b9022b77f866b59d"
format_stamp: "Formatted at 2026-09-28 06:40:16 on dist-test-slave-b28r"
server_key: "de32a52b5ba721aeec849ddf3ed73576"
server_key_iv: "2e897ae6751eaa64fc87bd50355002db"
server_key_version: "encryptionkey@0"
I20260928 06:40:16.473140 22164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "331ba69165b443a9b9022b77f866b59d"
format_stamp: "Formatted at 2026-09-28 06:40:16 on dist-test-slave-b28r"
server_key: "de32a52b5ba721aeec849ddf3ed73576"
server_key_iv: "2e897ae6751eaa64fc87bd50355002db"
server_key_version: "encryptionkey@0"
I20260928 06:40:16.474364 22164 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:40:16.475067 22179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:16.475241 22164 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:40:16.475360 22164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "331ba69165b443a9b9022b77f866b59d"
format_stamp: "Formatted at 2026-09-28 06:40:16 on dist-test-slave-b28r"
server_key: "de32a52b5ba721aeec849ddf3ed73576"
server_key_iv: "2e897ae6751eaa64fc87bd50355002db"
server_key_version: "encryptionkey@0"
I20260928 06:40:16.475437 22164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:16.493057 22164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:16.495182 22164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:16.495306 22164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:16.498900 22164 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:42181
I20260928 06:40:16.498939 22231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:42181 every 8 connection(s)
I20260928 06:40:16.499396 22164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:40:16.500012 22232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:16.502210 22232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 331ba69165b443a9b9022b77f866b59d: Bootstrap starting.
I20260928 06:40:16.502755 22232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 331ba69165b443a9b9022b77f866b59d: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:16.503006 22232 log.cc:824] T 00000000000000000000000000000000 P 331ba69165b443a9b9022b77f866b59d: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:16.503652 22232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 331ba69165b443a9b9022b77f866b59d: No bootstrap required, opened a new log
I20260928 06:40:16.503978 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 22164
I20260928 06:40:16.504084 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:40:16.504249 19277 external_mini_cluster.cc:1428] Setting key f4188f01718d0b84c6aeb7f514fd1f5c
I20260928 06:40:16.506156 22232 raft_consensus.cc:348] T 00000000000000000000000000000000 P 331ba69165b443a9b9022b77f866b59d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "331ba69165b443a9b9022b77f866b59d" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 42181 } }
I20260928 06:40:16.506331 22232 raft_consensus.cc:374] T 00000000000000000000000000000000 P 331ba69165b443a9b9022b77f866b59d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:16.506369 22232 raft_consensus.cc:729] T 00000000000000000000000000000000 P 331ba69165b443a9b9022b77f866b59d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 331ba69165b443a9b9022b77f866b59d, State: Initialized, Role: FOLLOWER
I20260928 06:40:16.506511 22232 consensus_queue.cc:260] T 00000000000000000000000000000000 P 331ba69165b443a9b9022b77f866b59d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "331ba69165b443a9b9022b77f866b59d" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 42181 } }
I20260928 06:40:16.506574 22232 raft_consensus.cc:388] T 00000000000000000000000000000000 P 331ba69165b443a9b9022b77f866b59d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:40:16.506594 22232 raft_consensus.cc:482] T 00000000000000000000000000000000 P 331ba69165b443a9b9022b77f866b59d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:40:16.506639 22232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 331ba69165b443a9b9022b77f866b59d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:16.507205 22232 raft_consensus.cc:504] T 00000000000000000000000000000000 P 331ba69165b443a9b9022b77f866b59d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "331ba69165b443a9b9022b77f866b59d" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 42181 } }
I20260928 06:40:16.507315 22232 leader_election.cc:302] T 00000000000000000000000000000000 P 331ba69165b443a9b9022b77f866b59d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 331ba69165b443a9b9022b77f866b59d; no voters:
I20260928 06:40:16.507457 22232 leader_election.cc:288] T 00000000000000000000000000000000 P 331ba69165b443a9b9022b77f866b59d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:40:16.507512 22237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 331ba69165b443a9b9022b77f866b59d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:16.507692 22232 sys_catalog.cc:564] T 00000000000000000000000000000000 P 331ba69165b443a9b9022b77f866b59d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:40:16.507690 22237 raft_consensus.cc:686] T 00000000000000000000000000000000 P 331ba69165b443a9b9022b77f866b59d [term 1 LEADER]: Becoming Leader. State: Replica: 331ba69165b443a9b9022b77f866b59d, State: Running, Role: LEADER
I20260928 06:40:16.507927 22237 consensus_queue.cc:237] T 00000000000000000000000000000000 P 331ba69165b443a9b9022b77f866b59d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "331ba69165b443a9b9022b77f866b59d" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 42181 } }
I20260928 06:40:16.508275 22238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 331ba69165b443a9b9022b77f866b59d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "331ba69165b443a9b9022b77f866b59d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "331ba69165b443a9b9022b77f866b59d" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 42181 } } }
I20260928 06:40:16.508348 22238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 331ba69165b443a9b9022b77f866b59d [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:16.508567 22239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 331ba69165b443a9b9022b77f866b59d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 331ba69165b443a9b9022b77f866b59d. Latest consensus state: current_term: 1 leader_uuid: "331ba69165b443a9b9022b77f866b59d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "331ba69165b443a9b9022b77f866b59d" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 42181 } } }
I20260928 06:40:16.508880 22239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 331ba69165b443a9b9022b77f866b59d [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:16.509030 22246 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:40:16.509807 22246 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 06:40:16 dist-test-slave-b28r krb5kdc[22151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:16.512046 22246 catalog_manager.cc:1347] Generated new cluster ID: 6e106f551c704462a354109b55a5b086
I20260928 06:40:16.512099 22246 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:40:16.515363 22236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:16.504694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48870 (local address 127.18.211.126:42181)
0928 06:40:16.504889 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:16.504900 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:16.504933 (+ 33us) server_negotiation.cc:407] Connection header received
0928 06:40:16.506127 (+ 1194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:16.506136 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:16.506200 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:16.506348 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:16.507037 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:16.508420 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:16.509443 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:16.509936 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:16.510042 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:16.512407 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:16.512411 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:16.512411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:16.512423 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:16.513829 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:16.514134 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:16.514137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:16.514138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:16.514228 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:16.514405 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:16.514405 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:16.514406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:16.514629 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:16.514935 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:16.515014 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 06:40:16.515173 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":86,"threads_started":1}
I20260928 06:40:16.516242 19277 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:40:16.527225 22246 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:40:16.528008 22246 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:40:16.529830 22236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:16.525693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48874 (local address 127.18.211.126:42181)
0928 06:40:16.525748 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:16.525750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:16.525874 (+ 124us) server_negotiation.cc:407] Connection header received
0928 06:40:16.525934 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:16.525935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:16.525946 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:16.525993 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:16.526354 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:16.527064 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:16.528195 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:16.528517 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:16.528588 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:16.529336 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:16.529338 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:16.529338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:40:16.529561 (+ 223us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:16.529580 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:40:16.529587 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:16.529649 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:16.529707 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 06:40:16.529747 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 06:40:16.534497 22246 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 331ba69165b443a9b9022b77f866b59d: Generated new TSK 0
I20260928 06:40:16.534675 22246 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:40:16.537114 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 22164
2026-09-28T06:40:16Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1208 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:40:16 dist-test-slave-b28r krb5kdc[22265](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:40:16 dist-test-slave-b28r krb5kdc[22265](info): set up 2 sockets
Sep 28 06:40:16 dist-test-slave-b28r krb5kdc[22265](info): commencing operation
krb5kdc: starting...
W20260928 06:40:17.483826 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:40:17 dist-test-slave-b28r krb5kdc[22265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:40:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:40:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.211.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
I20260928 06:40:17.590044 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:46195
--webserver_interface=127.18.211.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:34097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:46195
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:17.669632 22278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:17.669829 22278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:17.669850 22278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:17.670975 22278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:40:17.671016 22278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:17.671029 22278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:17.671041 22278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:40:17.671052 22278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:40:17.672576 22278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:34097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:46195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:46195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:17.672797 22278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:17.673000 22278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:17.677398 22286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:17.677439 22284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:17.677482 22283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:17.677759 22278 server_base.cc:1034] running on GCE node
I20260928 06:40:17.677975 22278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:17.678257 22278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:17.679400 22278 hybrid_clock.cc:648] HybridClock initialized: now 1790577617679383 us; error 26 us; skew 500 ppm
Sep 28 06:40:17 dist-test-slave-b28r krb5kdc[22265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577617, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:17.681766 22278 init.cc:377] Logged in from keytab as kudu/127.18.211.126@KRBTEST.COM (short username kudu)
I20260928 06:40:17.682196 22278 webserver.cc:460] Webserver started at http://127.18.211.126:38993/ using document root <none> and password file <none>
I20260928 06:40:17.682430 22278 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:17.682477 22278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:17.682608 22278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:17.683693 22278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "bcaf91a8ed9247d6881405734cd1d579"
format_stamp: "Formatted at 2026-09-28 06:40:17 on dist-test-slave-b28r"
server_key: "aea25eab62cbcbedf33a91dcaa59c4a6"
server_key_iv: "44b785f819892adb9aff6290046c3edb"
server_key_version: "encryptionkey@0"
I20260928 06:40:17.684015 22278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "bcaf91a8ed9247d6881405734cd1d579"
format_stamp: "Formatted at 2026-09-28 06:40:17 on dist-test-slave-b28r"
server_key: "aea25eab62cbcbedf33a91dcaa59c4a6"
server_key_iv: "44b785f819892adb9aff6290046c3edb"
server_key_version: "encryptionkey@0"
I20260928 06:40:17.685292 22278 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:40:17.686036 22293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:17.686197 22278 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:40:17.686280 22278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "bcaf91a8ed9247d6881405734cd1d579"
format_stamp: "Formatted at 2026-09-28 06:40:17 on dist-test-slave-b28r"
server_key: "aea25eab62cbcbedf33a91dcaa59c4a6"
server_key_iv: "44b785f819892adb9aff6290046c3edb"
server_key_version: "encryptionkey@0"
I20260928 06:40:17.686342 22278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:17.700237 22278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:17.702229 22278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:17.702376 22278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:17.706202 22278 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:46195
I20260928 06:40:17.706244 22345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:46195 every 8 connection(s)
I20260928 06:40:17.706596 22278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:40:17.707587 22346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:17.709828 22346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcaf91a8ed9247d6881405734cd1d579: Bootstrap starting.
I20260928 06:40:17.710371 22346 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcaf91a8ed9247d6881405734cd1d579: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:17.710616 22346 log.cc:824] T 00000000000000000000000000000000 P bcaf91a8ed9247d6881405734cd1d579: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:17.711294 22346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcaf91a8ed9247d6881405734cd1d579: No bootstrap required, opened a new log
I20260928 06:40:17.712809 22346 raft_consensus.cc:348] T 00000000000000000000000000000000 P bcaf91a8ed9247d6881405734cd1d579 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcaf91a8ed9247d6881405734cd1d579" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 46195 } }
I20260928 06:40:17.712944 22346 raft_consensus.cc:374] T 00000000000000000000000000000000 P bcaf91a8ed9247d6881405734cd1d579 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:17.712977 22346 raft_consensus.cc:729] T 00000000000000000000000000000000 P bcaf91a8ed9247d6881405734cd1d579 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcaf91a8ed9247d6881405734cd1d579, State: Initialized, Role: FOLLOWER
I20260928 06:40:17.713107 22346 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcaf91a8ed9247d6881405734cd1d579 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcaf91a8ed9247d6881405734cd1d579" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 46195 } }
I20260928 06:40:17.713163 22346 raft_consensus.cc:388] T 00000000000000000000000000000000 P bcaf91a8ed9247d6881405734cd1d579 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:40:17.713194 22346 raft_consensus.cc:482] T 00000000000000000000000000000000 P bcaf91a8ed9247d6881405734cd1d579 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:40:17.713239 22346 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bcaf91a8ed9247d6881405734cd1d579 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:17.713991 22346 raft_consensus.cc:504] T 00000000000000000000000000000000 P bcaf91a8ed9247d6881405734cd1d579 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcaf91a8ed9247d6881405734cd1d579" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 46195 } }
I20260928 06:40:17.714082 22346 leader_election.cc:302] T 00000000000000000000000000000000 P bcaf91a8ed9247d6881405734cd1d579 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bcaf91a8ed9247d6881405734cd1d579; no voters:
I20260928 06:40:17.714232 22346 leader_election.cc:288] T 00000000000000000000000000000000 P bcaf91a8ed9247d6881405734cd1d579 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:40:17.714273 22349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bcaf91a8ed9247d6881405734cd1d579 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:17.714421 22349 raft_consensus.cc:686] T 00000000000000000000000000000000 P bcaf91a8ed9247d6881405734cd1d579 [term 1 LEADER]: Becoming Leader. State: Replica: bcaf91a8ed9247d6881405734cd1d579, State: Running, Role: LEADER
I20260928 06:40:17.714432 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 22278
I20260928 06:40:17.714524 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:40:17.714511 22346 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcaf91a8ed9247d6881405734cd1d579 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:40:17.714547 22349 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcaf91a8ed9247d6881405734cd1d579 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcaf91a8ed9247d6881405734cd1d579" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 46195 } }
I20260928 06:40:17.714709 19277 external_mini_cluster.cc:1428] Setting key 8488748148e1e1c7d910bbf68073ee8c
I20260928 06:40:17.714882 22350 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcaf91a8ed9247d6881405734cd1d579 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bcaf91a8ed9247d6881405734cd1d579" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcaf91a8ed9247d6881405734cd1d579" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 46195 } } }
I20260928 06:40:17.714969 22350 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcaf91a8ed9247d6881405734cd1d579 [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:17.715265 22356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:40:17.715503 22349 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcaf91a8ed9247d6881405734cd1d579 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcaf91a8ed9247d6881405734cd1d579. Latest consensus state: current_term: 1 leader_uuid: "bcaf91a8ed9247d6881405734cd1d579" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcaf91a8ed9247d6881405734cd1d579" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 46195 } } }
I20260928 06:40:17.715576 22349 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcaf91a8ed9247d6881405734cd1d579 [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:17.716008 22356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:40:17.718283 22356 catalog_manager.cc:1347] Generated new cluster ID: ff5551af558d4d14b36a71acf9f2a57f
I20260928 06:40:17.718338 22356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:40:17 dist-test-slave-b28r krb5kdc[22265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:17.726368 22359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:17.715237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48224 (local address 127.18.211.126:46195)
0928 06:40:17.715681 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:17.715690 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:17.715955 (+ 265us) server_negotiation.cc:407] Connection header received
0928 06:40:17.716078 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:17.716085 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:17.716144 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:17.716274 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:17.717068 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:17.718160 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:17.719206 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:17.719849 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:17.719989 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:17.722642 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:17.722646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:17.722646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:17.722658 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:17.724561 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:17.724967 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:17.724970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:17.724970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:17.725068 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:17.725268 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:17.725269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:17.725269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:17.725644 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:17.725792 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:17.725926 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 06:40:17.726144 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":86,"threads_started":1}
I20260928 06:40:17.730687 22356 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:40:17.731235 22356 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:40:17.734593 22356 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bcaf91a8ed9247d6881405734cd1d579: Generated new TSK 0
I20260928 06:40:17.734758 22356 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.211.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab.
Entry for principal kudu/127.18.211.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab.
I20260928 06:40:17.757555 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.65:0
--local_ip_for_outbound_sockets=127.18.211.65
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:46195
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:34097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:17.836006 22372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:17.836210 22372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:17.836241 22372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:17.837366 22372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:17.837406 22372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:17.837440 22372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.65
I20260928 06:40:17.838857 22372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:34097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:46195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.65
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:17.839111 22372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:17.839320 22372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:17.844044 22380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:17.844117 22377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:17.844070 22378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:17.844247 22372 server_base.cc:1034] running on GCE node
I20260928 06:40:17.844524 22372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:17.844758 22372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:17.845907 22372 hybrid_clock.cc:648] HybridClock initialized: now 1790577617845890 us; error 34 us; skew 500 ppm
Sep 28 06:40:17 dist-test-slave-b28r krb5kdc[22265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577617, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:17.848439 22372 init.cc:377] Logged in from keytab as kudu/127.18.211.65@KRBTEST.COM (short username kudu)
I20260928 06:40:17.848836 22372 webserver.cc:460] Webserver started at http://127.18.211.65:35627/ using document root <none> and password file <none>
I20260928 06:40:17.849043 22372 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:17.849086 22372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:17.849200 22372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:17.850229 22372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-0/data/instance:
uuid: "b3e308d80dc9499eb465a7fc60a83964"
format_stamp: "Formatted at 2026-09-28 06:40:17 on dist-test-slave-b28r"
server_key: "749cf3a62f7395dc5d604cf22e3ed74f"
server_key_iv: "f2254e4b417f1bc0787e0fb5fba2ead4"
server_key_version: "encryptionkey@0"
I20260928 06:40:17.850545 22372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance:
uuid: "b3e308d80dc9499eb465a7fc60a83964"
format_stamp: "Formatted at 2026-09-28 06:40:17 on dist-test-slave-b28r"
server_key: "749cf3a62f7395dc5d604cf22e3ed74f"
server_key_iv: "f2254e4b417f1bc0787e0fb5fba2ead4"
server_key_version: "encryptionkey@0"
I20260928 06:40:17.851820 22372 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:40:17.852520 22387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:17.852656 22372 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:40:17.852728 22372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-0/wal
uuid: "b3e308d80dc9499eb465a7fc60a83964"
format_stamp: "Formatted at 2026-09-28 06:40:17 on dist-test-slave-b28r"
server_key: "749cf3a62f7395dc5d604cf22e3ed74f"
server_key_iv: "f2254e4b417f1bc0787e0fb5fba2ead4"
server_key_version: "encryptionkey@0"
I20260928 06:40:17.852792 22372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:17.863328 22372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:17.865823 22372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:17.865962 22372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:17.866183 22372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:17.866516 22372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:17.866562 22372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:17.866595 22372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:17.866613 22372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:17.872884 22372 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.65:33247
I20260928 06:40:17.872946 22500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.65:33247 every 8 connection(s)
I20260928 06:40:17.873219 22372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:40:17 dist-test-slave-b28r krb5kdc[22265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577617, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:17.881778 22359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:17.873882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:46113 (local address 127.18.211.126:46195)
0928 06:40:17.873922 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:17.873924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:17.874099 (+ 175us) server_negotiation.cc:407] Connection header received
0928 06:40:17.874602 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:17.874603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:17.874620 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:17.874671 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:17.875598 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:17.876109 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:17.877062 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:17.877258 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:17.877326 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:17.879283 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:17.879284 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:17.879284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:17.879291 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:17.880439 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:17.880840 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:17.880841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:17.880841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:17.880896 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:17.881109 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:17.881110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:17.881110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:17.881262 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:17.881319 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:17.881594 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 06:40:17.881684 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:40:17.881922 22503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:17.873853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:46195 (local address 127.18.211.65:46113)
0928 06:40:17.873987 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:17.874022 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:17.874506 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:17.874735 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:17.874739 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:17.875328 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:17.875508 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:17.875513 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:17.876190 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:17.876191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:17.876984 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:17.876987 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:17.877332 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:17.877333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:17.877445 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:17.877860 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:17.877868 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:17.879187 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:17.880527 (+ 1340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:17.880533 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:17.880535 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:17.880771 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:17.880951 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:17.880952 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:17.880952 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:17.881054 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:17.881324 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:17.881331 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:17.881461 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 06:40:17.881563 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 06:40:17.881721 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":66,"threads_started":1}
I20260928 06:40:17.882191 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 22372
I20260928 06:40:17.882300 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance
I20260928 06:40:17.882434 22501 heartbeater.cc:347] Connected to a master server at 127.18.211.126:46195
I20260928 06:40:17.882485 19277 external_mini_cluster.cc:1428] Setting key 5eb6d98c0559bff6774a66d80414fd65
I20260928 06:40:17.882529 22501 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:17.882694 22501 heartbeater.cc:511] Master 127.18.211.126:46195 requested a full tablet report, sending...
I20260928 06:40:17.883298 22310 ts_manager.cc:194] Registered new tserver with Master: b3e308d80dc9499eb465a7fc60a83964 (127.18.211.65:33247)
I20260928 06:40:17.883911 22310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.65@KRBTEST.COM'} at 127.18.211.65:46113
WARNING: no policy specified for kudu/127.18.211.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab.
Entry for principal kudu/127.18.211.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab.
I20260928 06:40:17.903949 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.66:0
--local_ip_for_outbound_sockets=127.18.211.66
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:46195
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:34097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:17.981971 22506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:17.982160 22506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:17.982182 22506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:17.983304 22506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:17.983341 22506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:17.983366 22506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.66
I20260928 06:40:17.984762 22506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:34097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:46195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.66
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:17.984939 22506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:17.985106 22506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:17.989766 22512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:17.989779 22511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:17.989758 22514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:17.990061 22506 server_base.cc:1034] running on GCE node
I20260928 06:40:17.990321 22506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:17.990552 22506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:17.991695 22506 hybrid_clock.cc:648] HybridClock initialized: now 1790577617991676 us; error 30 us; skew 500 ppm
Sep 28 06:40:17 dist-test-slave-b28r krb5kdc[22265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577617, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:17.994087 22506 init.cc:377] Logged in from keytab as kudu/127.18.211.66@KRBTEST.COM (short username kudu)
I20260928 06:40:17.994524 22506 webserver.cc:460] Webserver started at http://127.18.211.66:45239/ using document root <none> and password file <none>
I20260928 06:40:17.994722 22506 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:17.994771 22506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:17.994891 22506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:17.995926 22506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-1/data/instance:
uuid: "47ea8d6c045e43988fbb90f7d8420ff1"
format_stamp: "Formatted at 2026-09-28 06:40:17 on dist-test-slave-b28r"
server_key: "ab220c3dd30c7fb39497770d6509319e"
server_key_iv: "6c1af809468f91fd04caa1c5a0892f46"
server_key_version: "encryptionkey@0"
I20260928 06:40:17.996254 22506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance:
uuid: "47ea8d6c045e43988fbb90f7d8420ff1"
format_stamp: "Formatted at 2026-09-28 06:40:17 on dist-test-slave-b28r"
server_key: "ab220c3dd30c7fb39497770d6509319e"
server_key_iv: "6c1af809468f91fd04caa1c5a0892f46"
server_key_version: "encryptionkey@0"
I20260928 06:40:17.997481 22506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:40:17.998216 22521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:17.998392 22506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:40:17.998473 22506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-1/wal
uuid: "47ea8d6c045e43988fbb90f7d8420ff1"
format_stamp: "Formatted at 2026-09-28 06:40:17 on dist-test-slave-b28r"
server_key: "ab220c3dd30c7fb39497770d6509319e"
server_key_iv: "6c1af809468f91fd04caa1c5a0892f46"
server_key_version: "encryptionkey@0"
I20260928 06:40:17.998538 22506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:18.018615 22506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:18.020534 22506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:18.020673 22506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:18.020911 22506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:18.021225 22506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:18.021260 22506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:18.021287 22506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:18.021310 22506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:18.026556 22506 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.66:45271
I20260928 06:40:18.026623 22634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.66:45271 every 8 connection(s)
I20260928 06:40:18.026907 22506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
I20260928 06:40:18.028120 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 22506
I20260928 06:40:18.028211 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance
I20260928 06:40:18.028373 19277 external_mini_cluster.cc:1428] Setting key 81082617f9265599bebd5d274f231bb4
Sep 28 06:40:18 dist-test-slave-b28r krb5kdc[22265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577617, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.211.67@KRBTEST.COM; defaulting to no policy
I20260928 06:40:18.036260 22359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:18.027590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:38383 (local address 127.18.211.126:46195)
0928 06:40:18.027645 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:18.027647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:18.027852 (+ 205us) server_negotiation.cc:407] Connection header received
0928 06:40:18.028422 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:18.028423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:18.028436 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:18.028479 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:18.029489 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:18.029968 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:18.031022 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:18.031324 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:18.031391 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:18.033511 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:18.033512 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:18.033513 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:18.033519 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:18.034785 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:18.035369 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:18.035370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:18.035370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:18.035430 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:18.035718 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:18.035718 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:18.035719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:18.035837 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:18.035889 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:18.036131 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 06:40:18.036196 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:40:18.036440 22637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:18.027560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:46195 (local address 127.18.211.66:38383)
0928 06:40:18.027735 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:18.027771 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:18.028328 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:18.028566 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:18.028570 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:18.029200 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:18.029399 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:18.029403 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:18.030080 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:18.030081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:18.030939 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:18.030942 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:18.031407 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:18.031407 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:18.031511 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:18.031966 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:18.031975 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:18.033419 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:18.034869 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:18.034876 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:18.034879 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:18.035258 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:18.035498 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:18.035499 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:18.035500 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:18.035640 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:18.035900 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:18.035905 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:18.036027 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 06:40:18.036113 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 06:40:18.036233 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":94,"threads_started":1}
I20260928 06:40:18.036885 22635 heartbeater.cc:347] Connected to a master server at 127.18.211.126:46195
I20260928 06:40:18.037009 22635 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:18.037189 22635 heartbeater.cc:511] Master 127.18.211.126:46195 requested a full tablet report, sending...
I20260928 06:40:18.037596 22310 ts_manager.cc:194] Registered new tserver with Master: 47ea8d6c045e43988fbb90f7d8420ff1 (127.18.211.66:45271)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.67@KRBTEST.COM" created.
I20260928 06:40:18.037989 22310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.66@KRBTEST.COM'} at 127.18.211.66:38383
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab.
Entry for principal kudu/127.18.211.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab.
I20260928 06:40:18.049785 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.67:0
--local_ip_for_outbound_sockets=127.18.211.67
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:46195
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:34097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:18.128841 22640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:18.129035 22640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:18.129057 22640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:18.130163 22640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:18.130210 22640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:18.130236 22640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.67
I20260928 06:40:18.131669 22640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:34097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:46195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.67
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:18.131870 22640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:18.132081 22640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:18.136648 22645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:18.136647 22646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:18.136672 22648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:18.136899 22640 server_base.cc:1034] running on GCE node
I20260928 06:40:18.137141 22640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:18.137374 22640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:18.138552 22640 hybrid_clock.cc:648] HybridClock initialized: now 1790577618138531 us; error 29 us; skew 500 ppm
Sep 28 06:40:18 dist-test-slave-b28r krb5kdc[22265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577618, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:18.141031 22640 init.cc:377] Logged in from keytab as kudu/127.18.211.67@KRBTEST.COM (short username kudu)
I20260928 06:40:18.141435 22640 webserver.cc:460] Webserver started at http://127.18.211.67:36111/ using document root <none> and password file <none>
I20260928 06:40:18.141602 22640 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:18.141641 22640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:18.141726 22640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:18.142681 22640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-2/data/instance:
uuid: "4644d69bbf9b4fdd867f3406e2c795d0"
format_stamp: "Formatted at 2026-09-28 06:40:18 on dist-test-slave-b28r"
server_key: "51fdf23d22e7c723ff950f189d4f35ba"
server_key_iv: "e6d65ec316ef0f71ff6f8ec6b604c138"
server_key_version: "encryptionkey@0"
I20260928 06:40:18.142972 22640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance:
uuid: "4644d69bbf9b4fdd867f3406e2c795d0"
format_stamp: "Formatted at 2026-09-28 06:40:18 on dist-test-slave-b28r"
server_key: "51fdf23d22e7c723ff950f189d4f35ba"
server_key_iv: "e6d65ec316ef0f71ff6f8ec6b604c138"
server_key_version: "encryptionkey@0"
I20260928 06:40:18.144265 22640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:40:18.145027 22655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:18.145185 22640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:40:18.145267 22640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-2/wal
uuid: "4644d69bbf9b4fdd867f3406e2c795d0"
format_stamp: "Formatted at 2026-09-28 06:40:18 on dist-test-slave-b28r"
server_key: "51fdf23d22e7c723ff950f189d4f35ba"
server_key_iv: "e6d65ec316ef0f71ff6f8ec6b604c138"
server_key_version: "encryptionkey@0"
I20260928 06:40:18.145344 22640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:18.157248 22640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:18.159402 22640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:18.159530 22640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:18.159737 22640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:18.160041 22640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:18.160075 22640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:18.160109 22640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:18.160133 22640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:18.165512 22640 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.67:40215
I20260928 06:40:18.165575 22768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.67:40215 every 8 connection(s)
I20260928 06:40:18.165864 22640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:40:18 dist-test-slave-b28r krb5kdc[22265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577618, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:18.174247 22771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:18.166487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:46195 (local address 127.18.211.67:59525)
0928 06:40:18.166636 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:18.166671 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:18.167191 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:18.167406 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:18.167409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:18.167963 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:18.168147 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:18.168152 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:18.168830 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:18.168831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:18.169617 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:18.169619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:18.169955 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:18.169955 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:18.170055 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:18.170489 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:18.170497 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:18.171914 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:18.173153 (+ 1239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:18.173158 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:18.173159 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:18.173382 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:18.173530 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:18.173531 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:18.173531 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:18.173629 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:18.173804 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:18.173806 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:18.173911 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 06:40:18.173985 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 06:40:18.174094 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":70,"threads_started":1}
I20260928 06:40:18.174254 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 22640
I20260928 06:40:18.174226 22359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:18.166520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:59525 (local address 127.18.211.126:46195)
0928 06:40:18.166596 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:18.166598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:18.166851 (+ 253us) server_negotiation.cc:407] Connection header received
0928 06:40:18.167281 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:18.167282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:18.167293 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:18.167347 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:18.168231 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:18.168754 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:18.169697 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:18.169890 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:18.169949 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:18.171997 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:18.171998 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:18.171998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:18.172005 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:18.173086 (+ 1081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:18.173440 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:18.173441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:18.173441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:18.173485 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:18.173671 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:18.173671 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:18.173672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:18.173763 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:18.173805 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:18.174046 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 06:40:18.174135 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:40:18.174345 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance
I20260928 06:40:18.174511 19277 external_mini_cluster.cc:1428] Setting key 7bd7d81708cded09d5bf2532b7651f90
I20260928 06:40:18.174853 22769 heartbeater.cc:347] Connected to a master server at 127.18.211.126:46195
I20260928 06:40:18.174942 22769 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:18.175132 22769 heartbeater.cc:511] Master 127.18.211.126:46195 requested a full tablet report, sending...
I20260928 06:40:18.175477 22310 ts_manager.cc:194] Registered new tserver with Master: 4644d69bbf9b4fdd867f3406e2c795d0 (127.18.211.67:40215)
I20260928 06:40:18.175820 19277 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:40:18.175853 22310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.67@KRBTEST.COM'} at 127.18.211.67:59525
Sep 28 06:40:18 dist-test-slave-b28r krb5kdc[22265](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577618, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 06:40:18 dist-test-slave-b28r krb5kdc[22265](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577618, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:18.196795 22359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:18.190198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48228 (local address 127.18.211.126:46195)
0928 06:40:18.190293 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:18.190296 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:18.190405 (+ 109us) server_negotiation.cc:407] Connection header received
0928 06:40:18.190432 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:18.190433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:18.190444 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:18.190486 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:18.191011 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:18.191577 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:18.192292 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:18.192473 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:18.192524 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:18.194565 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:18.194566 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:18.194567 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:18.194572 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:18.195843 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:18.196154 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:18.196155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:18.196155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:18.196219 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:18.196365 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:18.196366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:18.196366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:18.196497 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:18.196567 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:18.196647 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 06:40:18.196710 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260928 06:40:18.197026 22310 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48228
I20260928 06:40:18.197487 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 22372
I20260928 06:40:18.201435 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 22506
I20260928 06:40:18.205503 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 22640
I20260928 06:40:18.209316 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 22278
2026-09-28T06:40:18Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1673 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:40:18 dist-test-slave-b28r krb5kdc[22781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:40:18 dist-test-slave-b28r krb5kdc[22781](info): set up 2 sockets
Sep 28 06:40:18 dist-test-slave-b28r krb5kdc[22781](info): commencing operation
krb5kdc: starting...
W20260928 06:40:19.159850 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:40:19 dist-test-slave-b28r krb5kdc[22781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:40:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:40:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.211.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
I20260928 06:40:19.269747 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:35785
--webserver_interface=127.18.211.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:42771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:35785
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:19.347731 22794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:19.347934 22794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:19.347958 22794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:19.349082 22794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:40:19.349123 22794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:19.349135 22794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:19.349146 22794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:40:19.349157 22794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:40:19.350610 22794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:42771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:35785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:35785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:19.350816 22794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:19.351020 22794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:19.355523 22799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:19.355587 22800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:19.355660 22802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:19.355667 22794 server_base.cc:1034] running on GCE node
I20260928 06:40:19.355983 22794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:19.356220 22794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:19.357355 22794 hybrid_clock.cc:648] HybridClock initialized: now 1790577619357345 us; error 27 us; skew 500 ppm
Sep 28 06:40:19 dist-test-slave-b28r krb5kdc[22781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577619, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:19.359746 22794 init.cc:377] Logged in from keytab as kudu/127.18.211.126@KRBTEST.COM (short username kudu)
I20260928 06:40:19.360162 22794 webserver.cc:460] Webserver started at http://127.18.211.126:42959/ using document root <none> and password file <none>
I20260928 06:40:19.360378 22794 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:19.360426 22794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:19.360550 22794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:19.361640 22794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "2e1d9d956b684108b2e949ffabce982d"
format_stamp: "Formatted at 2026-09-28 06:40:19 on dist-test-slave-b28r"
server_key: "689406ebd06d1a29f97c2fa323b00718"
server_key_iv: "59662b42f68f7814791875118f66ceba"
server_key_version: "encryptionkey@0"
I20260928 06:40:19.361965 22794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "2e1d9d956b684108b2e949ffabce982d"
format_stamp: "Formatted at 2026-09-28 06:40:19 on dist-test-slave-b28r"
server_key: "689406ebd06d1a29f97c2fa323b00718"
server_key_iv: "59662b42f68f7814791875118f66ceba"
server_key_version: "encryptionkey@0"
I20260928 06:40:19.363255 22794 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:40:19.363938 22809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:19.364092 22794 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:19.364171 22794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "2e1d9d956b684108b2e949ffabce982d"
format_stamp: "Formatted at 2026-09-28 06:40:19 on dist-test-slave-b28r"
server_key: "689406ebd06d1a29f97c2fa323b00718"
server_key_iv: "59662b42f68f7814791875118f66ceba"
server_key_version: "encryptionkey@0"
I20260928 06:40:19.364235 22794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:19.375958 22794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:19.377898 22794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:19.378022 22794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:19.381811 22794 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:35785
I20260928 06:40:19.381877 22861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:35785 every 8 connection(s)
I20260928 06:40:19.382304 22794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:40:19.382907 22862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:19.384075 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 22794
I20260928 06:40:19.384166 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:40:19.384356 19277 external_mini_cluster.cc:1428] Setting key 42be2cc1fa473003d3560589099a2d32
I20260928 06:40:19.385448 22862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e1d9d956b684108b2e949ffabce982d: Bootstrap starting.
I20260928 06:40:19.386091 22862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e1d9d956b684108b2e949ffabce982d: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:19.386371 22862 log.cc:824] T 00000000000000000000000000000000 P 2e1d9d956b684108b2e949ffabce982d: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:19.387032 22862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e1d9d956b684108b2e949ffabce982d: No bootstrap required, opened a new log
I20260928 06:40:19.388803 22862 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e1d9d956b684108b2e949ffabce982d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1d9d956b684108b2e949ffabce982d" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35785 } }
I20260928 06:40:19.388933 22862 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e1d9d956b684108b2e949ffabce982d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:19.388964 22862 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e1d9d956b684108b2e949ffabce982d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e1d9d956b684108b2e949ffabce982d, State: Initialized, Role: FOLLOWER
I20260928 06:40:19.389083 22862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e1d9d956b684108b2e949ffabce982d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1d9d956b684108b2e949ffabce982d" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35785 } }
I20260928 06:40:19.389155 22862 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e1d9d956b684108b2e949ffabce982d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:40:19.389186 22862 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e1d9d956b684108b2e949ffabce982d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:40:19.389233 22862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e1d9d956b684108b2e949ffabce982d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:19.389739 22862 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e1d9d956b684108b2e949ffabce982d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1d9d956b684108b2e949ffabce982d" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35785 } }
I20260928 06:40:19.389855 22862 leader_election.cc:302] T 00000000000000000000000000000000 P 2e1d9d956b684108b2e949ffabce982d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e1d9d956b684108b2e949ffabce982d; no voters:
I20260928 06:40:19.390005 22862 leader_election.cc:288] T 00000000000000000000000000000000 P 2e1d9d956b684108b2e949ffabce982d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:40:19.390043 22867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e1d9d956b684108b2e949ffabce982d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:19.390232 22867 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e1d9d956b684108b2e949ffabce982d [term 1 LEADER]: Becoming Leader. State: Replica: 2e1d9d956b684108b2e949ffabce982d, State: Running, Role: LEADER
I20260928 06:40:19.390250 22862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e1d9d956b684108b2e949ffabce982d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:40:19.390352 22867 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e1d9d956b684108b2e949ffabce982d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1d9d956b684108b2e949ffabce982d" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35785 } }
Sep 28 06:40:19 dist-test-slave-b28r krb5kdc[22781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:19.390734 22868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e1d9d956b684108b2e949ffabce982d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e1d9d956b684108b2e949ffabce982d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1d9d956b684108b2e949ffabce982d" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35785 } } }
I20260928 06:40:19.390815 22868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e1d9d956b684108b2e949ffabce982d [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:19.390978 22869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e1d9d956b684108b2e949ffabce982d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e1d9d956b684108b2e949ffabce982d. Latest consensus state: current_term: 1 leader_uuid: "2e1d9d956b684108b2e949ffabce982d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e1d9d956b684108b2e949ffabce982d" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35785 } } }
I20260928 06:40:19.391067 22869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e1d9d956b684108b2e949ffabce982d [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:19.391481 22874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:40:19.391973 22874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:40:19.393488 22874 catalog_manager.cc:1347] Generated new cluster ID: f34428d1267e4660af3d6c5672435380
I20260928 06:40:19.393536 22874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:40:19.394482 22865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:19.384820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57852 (local address 127.18.211.126:35785)
0928 06:40:19.384968 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:19.384975 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:19.384998 (+ 23us) server_negotiation.cc:407] Connection header received
0928 06:40:19.385388 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:19.385394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:19.385443 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:19.385573 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:19.386162 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:19.387233 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:19.387947 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:19.388375 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:19.388469 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:19.390947 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:19.390951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:19.390952 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:19.390964 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:19.392926 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:19.393309 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:19.393312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:19.393313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:19.393411 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:19.393606 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:19.393606 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:19.393607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:19.393885 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:19.394025 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:19.394154 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 06:40:19.394297 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
I20260928 06:40:19.398391 22874 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:40:19.399073 22874 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:40:19.405478 22874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e1d9d956b684108b2e949ffabce982d: Generated new TSK 0
I20260928 06:40:19.405629 22874 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.211.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab.
Entry for principal kudu/127.18.211.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab.
I20260928 06:40:19.433449 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.65:0
--local_ip_for_outbound_sockets=127.18.211.65
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35785
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:42771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:19.512189 22888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:19.512398 22888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:19.512420 22888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:19.513558 22888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:19.513602 22888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:19.513626 22888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.65
I20260928 06:40:19.515177 22888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:42771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.65
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:19.515405 22888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:19.515625 22888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:19.520306 22893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:19.520344 22896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:19.520308 22894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:19.520532 22888 server_base.cc:1034] running on GCE node
I20260928 06:40:19.520756 22888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:19.520972 22888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:19.522105 22888 hybrid_clock.cc:648] HybridClock initialized: now 1790577619522093 us; error 29 us; skew 500 ppm
Sep 28 06:40:19 dist-test-slave-b28r krb5kdc[22781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577619, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:19.524607 22888 init.cc:377] Logged in from keytab as kudu/127.18.211.65@KRBTEST.COM (short username kudu)
I20260928 06:40:19.525060 22888 webserver.cc:460] Webserver started at http://127.18.211.65:33373/ using document root <none> and password file <none>
I20260928 06:40:19.525265 22888 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:19.525309 22888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:19.525415 22888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:19.526463 22888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-0/data/instance:
uuid: "e8c7edd4c4104802b8f50a2ef078f6a2"
format_stamp: "Formatted at 2026-09-28 06:40:19 on dist-test-slave-b28r"
server_key: "f6632bb29e4acf3efd9aaf1c45fc527b"
server_key_iv: "3c68a7d56e380ad1cdd799464d180b77"
server_key_version: "encryptionkey@0"
I20260928 06:40:19.526789 22888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance:
uuid: "e8c7edd4c4104802b8f50a2ef078f6a2"
format_stamp: "Formatted at 2026-09-28 06:40:19 on dist-test-slave-b28r"
server_key: "f6632bb29e4acf3efd9aaf1c45fc527b"
server_key_iv: "3c68a7d56e380ad1cdd799464d180b77"
server_key_version: "encryptionkey@0"
I20260928 06:40:19.528172 22888 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:40:19.528865 22903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:19.528987 22888 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:40:19.529071 22888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-0/wal
uuid: "e8c7edd4c4104802b8f50a2ef078f6a2"
format_stamp: "Formatted at 2026-09-28 06:40:19 on dist-test-slave-b28r"
server_key: "f6632bb29e4acf3efd9aaf1c45fc527b"
server_key_iv: "3c68a7d56e380ad1cdd799464d180b77"
server_key_version: "encryptionkey@0"
I20260928 06:40:19.529139 22888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:19.539551 22888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:19.541610 22888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:19.541733 22888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:19.541947 22888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:19.542236 22888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:19.542269 22888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:19.542296 22888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:19.542321 22888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:19.548026 22888 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.65:36181
I20260928 06:40:19.548079 23016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.65:36181 every 8 connection(s)
I20260928 06:40:19.548458 22888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:40:19 dist-test-slave-b28r krb5kdc[22781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577619, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:19.557293 22865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:19.549156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:56773 (local address 127.18.211.126:35785)
0928 06:40:19.549243 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:19.549245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:19.549424 (+ 179us) server_negotiation.cc:407] Connection header received
0928 06:40:19.550386 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:19.550387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:19.550399 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:19.550469 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:19.551186 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:19.551693 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:19.552699 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:19.552922 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:19.552978 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:19.555060 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:19.555061 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:19.555062 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:19.555068 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:19.556242 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:19.556582 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:19.556582 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:19.556583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:19.556632 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:19.556820 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:19.556821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:19.556821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:19.556916 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:19.556961 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:19.557159 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 06:40:19.557221 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 06:40:19.557458 23019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:19.549094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35785 (local address 127.18.211.65:56773)
0928 06:40:19.549276 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:19.549317 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:19.550273 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:19.550541 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:19.550546 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:19.550849 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:19.551042 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:19.551048 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:19.551804 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:19.551805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:19.552614 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:19.552616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:19.553014 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:19.553015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:19.553128 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:19.553626 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:19.553634 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:19.554979 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:19.556315 (+ 1336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:19.556321 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:19.556322 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:19.556524 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:19.556675 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:19.556676 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:19.556676 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:19.556764 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:19.556956 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:19.556959 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:19.557067 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 06:40:19.557142 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 06:40:19.557257 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":93,"threads_started":1}
I20260928 06:40:19.557789 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 22888
I20260928 06:40:19.557896 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance
I20260928 06:40:19.557947 23017 heartbeater.cc:347] Connected to a master server at 127.18.211.126:35785
I20260928 06:40:19.558048 23017 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:19.558070 19277 external_mini_cluster.cc:1428] Setting key dc490198b460e514d7b085366fd67851
I20260928 06:40:19.558220 23017 heartbeater.cc:511] Master 127.18.211.126:35785 requested a full tablet report, sending...
I20260928 06:40:19.558735 22825 ts_manager.cc:194] Registered new tserver with Master: e8c7edd4c4104802b8f50a2ef078f6a2 (127.18.211.65:36181)
I20260928 06:40:19.559432 22825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.65@KRBTEST.COM'} at 127.18.211.65:56773
WARNING: no policy specified for kudu/127.18.211.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab.
Entry for principal kudu/127.18.211.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab.
I20260928 06:40:19.581450 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.66:0
--local_ip_for_outbound_sockets=127.18.211.66
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35785
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:42771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:19.660938 23022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:19.661140 23022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:19.661171 23022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:19.662360 23022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:19.662405 23022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:19.662439 23022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.66
I20260928 06:40:19.663936 23022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:42771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.66
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:19.664168 23022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:19.664376 23022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:19.669200 23030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:19.669185 23028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:19.669277 23022 server_base.cc:1034] running on GCE node
W20260928 06:40:19.669176 23027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:19.669611 23022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:19.669857 23022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:19.670997 23022 hybrid_clock.cc:648] HybridClock initialized: now 1790577619670991 us; error 34 us; skew 500 ppm
Sep 28 06:40:19 dist-test-slave-b28r krb5kdc[22781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577619, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:19.673432 23022 init.cc:377] Logged in from keytab as kudu/127.18.211.66@KRBTEST.COM (short username kudu)
I20260928 06:40:19.673883 23022 webserver.cc:460] Webserver started at http://127.18.211.66:44293/ using document root <none> and password file <none>
I20260928 06:40:19.674106 23022 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:19.674161 23022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:19.674293 23022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:19.675367 23022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-1/data/instance:
uuid: "00f1bbc72d35439f9b91274aee709bb8"
format_stamp: "Formatted at 2026-09-28 06:40:19 on dist-test-slave-b28r"
server_key: "4a1154dbfe688a9ab1178b208c598155"
server_key_iv: "22a20362231c81d38b45e2436a414d8a"
server_key_version: "encryptionkey@0"
I20260928 06:40:19.675743 23022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance:
uuid: "00f1bbc72d35439f9b91274aee709bb8"
format_stamp: "Formatted at 2026-09-28 06:40:19 on dist-test-slave-b28r"
server_key: "4a1154dbfe688a9ab1178b208c598155"
server_key_iv: "22a20362231c81d38b45e2436a414d8a"
server_key_version: "encryptionkey@0"
I20260928 06:40:19.677030 23022 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:40:19.677750 23037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:19.677907 23022 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:40:19.677985 23022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-1/wal
uuid: "00f1bbc72d35439f9b91274aee709bb8"
format_stamp: "Formatted at 2026-09-28 06:40:19 on dist-test-slave-b28r"
server_key: "4a1154dbfe688a9ab1178b208c598155"
server_key_iv: "22a20362231c81d38b45e2436a414d8a"
server_key_version: "encryptionkey@0"
I20260928 06:40:19.678052 23022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:19.707674 23022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:19.710004 23022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:19.710161 23022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:19.710407 23022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:19.710722 23022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:19.710760 23022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:19.710793 23022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:19.710820 23022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:19.716928 23022 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.66:38619
I20260928 06:40:19.717000 23150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.66:38619 every 8 connection(s)
I20260928 06:40:19.717295 23022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:40:19 dist-test-slave-b28r krb5kdc[22781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577619, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:19.725626 22865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:19.718017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:36829 (local address 127.18.211.126:35785)
0928 06:40:19.718071 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:19.718073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:19.718265 (+ 192us) server_negotiation.cc:407] Connection header received
0928 06:40:19.719210 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:19.719212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:19.719222 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:19.719291 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:19.719891 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:19.720396 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:19.721370 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:19.721546 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:19.721590 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:19.723577 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:19.723578 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:19.723578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:19.723585 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:19.724624 (+ 1039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:19.724961 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:19.724962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:19.724962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:19.725008 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:19.725161 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:19.725162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:19.725162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:19.725248 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:19.725291 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:19.725504 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 06:40:19.725561 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:40:19.725790 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:19.717969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35785 (local address 127.18.211.66:36829)
0928 06:40:19.718127 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:19.718179 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:19.719066 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:19.719355 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:19.719359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:19.719610 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:19.719811 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:19.719816 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:19.720483 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:19.720484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:19.721301 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:19.721304 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:19.721628 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:19.721629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:19.721735 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:19.722190 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:19.722198 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:19.723494 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:19.724686 (+ 1192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:19.724691 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:19.724693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:19.724906 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:19.725044 (+ 138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:19.725044 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:19.725044 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:19.725124 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:19.725294 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:19.725298 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:19.725401 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 06:40:19.725478 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 06:40:19.725587 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
I20260928 06:40:19.726179 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 23022
I20260928 06:40:19.726255 23151 heartbeater.cc:347] Connected to a master server at 127.18.211.126:35785
I20260928 06:40:19.726285 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance
I20260928 06:40:19.726351 23151 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:19.726473 19277 external_mini_cluster.cc:1428] Setting key 603b7ef1d442a0b09b3da10aa673ab7f
I20260928 06:40:19.726513 23151 heartbeater.cc:511] Master 127.18.211.126:35785 requested a full tablet report, sending...
I20260928 06:40:19.726886 22826 ts_manager.cc:194] Registered new tserver with Master: 00f1bbc72d35439f9b91274aee709bb8 (127.18.211.66:38619)
I20260928 06:40:19.727339 22826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.66@KRBTEST.COM'} at 127.18.211.66:36829
WARNING: no policy specified for kudu/127.18.211.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab.
Entry for principal kudu/127.18.211.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab.
I20260928 06:40:19.747284 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.67:0
--local_ip_for_outbound_sockets=127.18.211.67
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35785
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:42771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:19.826632 23156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:19.826848 23156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:19.826880 23156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:19.828076 23156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:19.828126 23156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:19.828172 23156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.67
I20260928 06:40:19.829671 23156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:42771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.67
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:19.829921 23156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:19.830155 23156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:19.834882 23162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:19.834930 23164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:19.834882 23161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:19.835285 23156 server_base.cc:1034] running on GCE node
I20260928 06:40:19.835506 23156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:19.835732 23156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:19.836870 23156 hybrid_clock.cc:648] HybridClock initialized: now 1790577619836853 us; error 36 us; skew 500 ppm
Sep 28 06:40:19 dist-test-slave-b28r krb5kdc[22781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577619, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:19.841588 23156 init.cc:377] Logged in from keytab as kudu/127.18.211.67@KRBTEST.COM (short username kudu)
I20260928 06:40:19.842023 23156 webserver.cc:460] Webserver started at http://127.18.211.67:37693/ using document root <none> and password file <none>
I20260928 06:40:19.842228 23156 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:19.842314 23156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:19.842434 23156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:19.843485 23156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-2/data/instance:
uuid: "381d539306dd404c92e9c420e219300e"
format_stamp: "Formatted at 2026-09-28 06:40:19 on dist-test-slave-b28r"
server_key: "f79a2b7e3c6ab715d7b41b0b92261b18"
server_key_iv: "ded03b8c557a55441f7b24fc779421c8"
server_key_version: "encryptionkey@0"
I20260928 06:40:19.843794 23156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance:
uuid: "381d539306dd404c92e9c420e219300e"
format_stamp: "Formatted at 2026-09-28 06:40:19 on dist-test-slave-b28r"
server_key: "f79a2b7e3c6ab715d7b41b0b92261b18"
server_key_iv: "ded03b8c557a55441f7b24fc779421c8"
server_key_version: "encryptionkey@0"
I20260928 06:40:19.844997 23156 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:40:19.845733 23171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:19.845901 23156 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:40:19.845981 23156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-2/wal
uuid: "381d539306dd404c92e9c420e219300e"
format_stamp: "Formatted at 2026-09-28 06:40:19 on dist-test-slave-b28r"
server_key: "f79a2b7e3c6ab715d7b41b0b92261b18"
server_key_iv: "ded03b8c557a55441f7b24fc779421c8"
server_key_version: "encryptionkey@0"
I20260928 06:40:19.846038 23156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:19.858119 23156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:19.859937 23156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:19.860067 23156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:19.860268 23156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:19.860548 23156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:19.860580 23156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:19.860608 23156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:19.860628 23156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:19.866415 23156 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.67:41453
I20260928 06:40:19.866505 23284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.67:41453 every 8 connection(s)
I20260928 06:40:19.866822 23156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
I20260928 06:40:19.871528 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 23156
I20260928 06:40:19.871622 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance
I20260928 06:40:19.871793 19277 external_mini_cluster.cc:1428] Setting key ddb0015416409d3ffd9e3121b80c3132
Sep 28 06:40:19 dist-test-slave-b28r krb5kdc[22781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577619, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:19.875933 22865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:19.867572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:42631 (local address 127.18.211.126:35785)
0928 06:40:19.867633 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:19.867635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:19.867859 (+ 224us) server_negotiation.cc:407] Connection header received
0928 06:40:19.868855 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:19.868856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:19.868867 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:19.868925 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:19.869557 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:19.870056 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:19.871060 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:19.871392 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:19.871466 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:19.873519 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:19.873521 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:19.873521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:19.873527 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:19.874664 (+ 1137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:19.875147 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:19.875148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:19.875148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:19.875209 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:19.875457 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:19.875458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:19.875458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:19.875563 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:19.875608 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:19.875815 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 06:40:19.875872 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 06:40:19.876098 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:19.867529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35785 (local address 127.18.211.67:42631)
0928 06:40:19.867740 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:19.867781 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:19.868692 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:19.868997 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:19.869001 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:19.869268 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:19.869451 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:19.869455 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:19.870153 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:19.870154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:19.870979 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:19.870981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:19.871488 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:19.871489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:19.871615 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:19.872141 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:19.872149 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:19.873440 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:19.874785 (+ 1345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:19.874791 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:19.874793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:19.875056 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:19.875300 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:19.875301 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:19.875302 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:19.875408 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:19.875605 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:19.875608 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:19.875713 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 06:40:19.875799 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 06:40:19.875912 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":95,"threads_started":1}
I20260928 06:40:19.876543 23285 heartbeater.cc:347] Connected to a master server at 127.18.211.126:35785
I20260928 06:40:19.876626 23285 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:19.876794 23285 heartbeater.cc:511] Master 127.18.211.126:35785 requested a full tablet report, sending...
I20260928 06:40:19.877116 22826 ts_manager.cc:194] Registered new tserver with Master: 381d539306dd404c92e9c420e219300e (127.18.211.67:41453)
I20260928 06:40:19.877460 22826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.67@KRBTEST.COM'} at 127.18.211.67:42631
I20260928 06:40:19.883324 19277 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 06:40:19 dist-test-slave-b28r krb5kdc[22781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:40:19 dist-test-slave-b28r krb5kdc[22781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.65@KRBTEST.COM
I20260928 06:40:19.906720 23292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:19.899486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43288 (local address 127.18.211.65:36181)
0928 06:40:19.899654 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:19.899659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:19.899669 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:40:19.899694 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:19.899696 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:19.899710 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:19.899774 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:19.900333 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:19.901024 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:19.901690 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:19.901992 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:19.902039 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:19.904158 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:19.904166 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:19.904166 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:19.904172 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:19.905674 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:19.905985 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:19.905989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:19.905989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:19.906042 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:19.906201 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:19.906202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:19.906203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:19.906339 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:19.906464 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:19.906541 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 06:40:19.906616 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":99,"threads_started":1}
Sep 28 06:40:19 dist-test-slave-b28r krb5kdc[22781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577619, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:19.914127 22865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:19.907828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57866 (local address 127.18.211.126:35785)
0928 06:40:19.907878 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:19.907880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:19.907903 (+ 23us) server_negotiation.cc:407] Connection header received
0928 06:40:19.907982 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:19.907983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:19.907994 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:19.908051 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:19.908556 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:19.909039 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:19.909709 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:19.909892 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:19.909949 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:19.911959 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:19.911960 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:19.911961 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:19.911969 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:19.913092 (+ 1123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:19.913422 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:19.913424 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:19.913424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:19.913475 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:19.913626 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:19.913627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:19.913627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:19.913776 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:19.913861 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:19.913969 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 06:40:19.914040 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260928 06:40:19.914330 22826 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57866
I20260928 06:40:19.914695 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 22888
I20260928 06:40:19.918888 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 23022
I20260928 06:40:19.922801 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 23156
I20260928 06:40:19.926501 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 22794
2026-09-28T06:40:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1717 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:40:19 dist-test-slave-b28r krb5kdc[23297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:40:19 dist-test-slave-b28r krb5kdc[23297](info): set up 2 sockets
Sep 28 06:40:19 dist-test-slave-b28r krb5kdc[23297](info): commencing operation
krb5kdc: starting...
W20260928 06:40:20.863695 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.918s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:40:20 dist-test-slave-b28r krb5kdc[23297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:40:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:40:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.211.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
I20260928 06:40:20.971448 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:35449
--webserver_interface=127.18.211.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:37251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:35449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:21.050024 23310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:21.050232 23310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:21.050261 23310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:21.051447 23310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:40:21.051491 23310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:21.051512 23310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:21.051527 23310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:40:21.051542 23310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:40:21.053084 23310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:37251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:35449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:35449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:21.053334 23310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:21.053540 23310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:21.058182 23315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:21.058185 23318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:21.058171 23316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:21.058416 23310 server_base.cc:1034] running on GCE node
I20260928 06:40:21.058714 23310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:21.058993 23310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:21.060134 23310 hybrid_clock.cc:648] HybridClock initialized: now 1790577621060118 us; error 32 us; skew 500 ppm
Sep 28 06:40:21 dist-test-slave-b28r krb5kdc[23297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577621, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:21.062597 23310 init.cc:377] Logged in from keytab as kudu/127.18.211.126@KRBTEST.COM (short username kudu)
I20260928 06:40:21.062934 23310 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:21.062983 23310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:21.063076 23310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:21.064113 23310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "a6a4afc02b974ef99b8ff285bb5487ff"
format_stamp: "Formatted at 2026-09-28 06:40:21 on dist-test-slave-b28r"
server_key: "de9fb18d95f6cb9cc916fa57962b226f"
server_key_iv: "f96063b3a0b6916ed6f3b3282ecfa0bf"
server_key_version: "encryptionkey@0"
I20260928 06:40:21.064469 23310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "a6a4afc02b974ef99b8ff285bb5487ff"
format_stamp: "Formatted at 2026-09-28 06:40:21 on dist-test-slave-b28r"
server_key: "de9fb18d95f6cb9cc916fa57962b226f"
server_key_iv: "f96063b3a0b6916ed6f3b3282ecfa0bf"
server_key_version: "encryptionkey@0"
I20260928 06:40:21.065644 23310 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:40:21.066325 23324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:21.066514 23310 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:21.066602 23310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "a6a4afc02b974ef99b8ff285bb5487ff"
format_stamp: "Formatted at 2026-09-28 06:40:21 on dist-test-slave-b28r"
server_key: "de9fb18d95f6cb9cc916fa57962b226f"
server_key_iv: "f96063b3a0b6916ed6f3b3282ecfa0bf"
server_key_version: "encryptionkey@0"
I20260928 06:40:21.066670 23310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:21.073660 23310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:21.075758 23310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:21.075886 23310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:21.079515 23310 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:35449
I20260928 06:40:21.079541 23376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:35449 every 8 connection(s)
I20260928 06:40:21.079818 23310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:40:21.080785 23377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:21.083051 23377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6a4afc02b974ef99b8ff285bb5487ff: Bootstrap starting.
I20260928 06:40:21.083652 23377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6a4afc02b974ef99b8ff285bb5487ff: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:21.083880 23377 log.cc:824] T 00000000000000000000000000000000 P a6a4afc02b974ef99b8ff285bb5487ff: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:21.084520 23377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6a4afc02b974ef99b8ff285bb5487ff: No bootstrap required, opened a new log
I20260928 06:40:21.085824 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 23310
I20260928 06:40:21.085922 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:40:21.086081 19277 external_mini_cluster.cc:1428] Setting key f4b59ba7bfdce1b6e33cd07dbc010845
I20260928 06:40:21.086100 23377 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6a4afc02b974ef99b8ff285bb5487ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a4afc02b974ef99b8ff285bb5487ff" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35449 } }
I20260928 06:40:21.086264 23377 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6a4afc02b974ef99b8ff285bb5487ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:21.086313 23377 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6a4afc02b974ef99b8ff285bb5487ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a4afc02b974ef99b8ff285bb5487ff, State: Initialized, Role: FOLLOWER
I20260928 06:40:21.086462 23377 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6a4afc02b974ef99b8ff285bb5487ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a4afc02b974ef99b8ff285bb5487ff" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35449 } }
I20260928 06:40:21.086544 23377 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6a4afc02b974ef99b8ff285bb5487ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:40:21.086575 23377 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6a4afc02b974ef99b8ff285bb5487ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:40:21.086624 23377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6a4afc02b974ef99b8ff285bb5487ff [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:21.087692 23377 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6a4afc02b974ef99b8ff285bb5487ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a4afc02b974ef99b8ff285bb5487ff" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35449 } }
I20260928 06:40:21.087806 23377 leader_election.cc:302] T 00000000000000000000000000000000 P a6a4afc02b974ef99b8ff285bb5487ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6a4afc02b974ef99b8ff285bb5487ff; no voters:
I20260928 06:40:21.087999 23377 leader_election.cc:288] T 00000000000000000000000000000000 P a6a4afc02b974ef99b8ff285bb5487ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:40:21.088035 23382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6a4afc02b974ef99b8ff285bb5487ff [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:21.088177 23382 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6a4afc02b974ef99b8ff285bb5487ff [term 1 LEADER]: Becoming Leader. State: Replica: a6a4afc02b974ef99b8ff285bb5487ff, State: Running, Role: LEADER
I20260928 06:40:21.088287 23377 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6a4afc02b974ef99b8ff285bb5487ff [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:40:21.088297 23382 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6a4afc02b974ef99b8ff285bb5487ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a4afc02b974ef99b8ff285bb5487ff" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35449 } }
I20260928 06:40:21.088796 23384 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6a4afc02b974ef99b8ff285bb5487ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6a4afc02b974ef99b8ff285bb5487ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a4afc02b974ef99b8ff285bb5487ff" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35449 } } }
I20260928 06:40:21.088812 23383 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6a4afc02b974ef99b8ff285bb5487ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6a4afc02b974ef99b8ff285bb5487ff. Latest consensus state: current_term: 1 leader_uuid: "a6a4afc02b974ef99b8ff285bb5487ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a4afc02b974ef99b8ff285bb5487ff" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35449 } } }
I20260928 06:40:21.088958 23384 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6a4afc02b974ef99b8ff285bb5487ff [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:21.088968 23383 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6a4afc02b974ef99b8ff285bb5487ff [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:21.089553 23391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:40:21.090029 23391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:40:21.091737 23391 catalog_manager.cc:1347] Generated new cluster ID: d8c8e3d4fb314d019e6e3108ca9574aa
I20260928 06:40:21.091785 23391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:40:21 dist-test-slave-b28r krb5kdc[23297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:21.098030 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:21.086549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40666 (local address 127.18.211.126:35449)
0928 06:40:21.086724 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:21.086735 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:21.086846 (+ 111us) server_negotiation.cc:407] Connection header received
0928 06:40:21.086992 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:21.086998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:21.087045 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:21.087182 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:21.087985 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.089431 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.090439 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.091042 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.091191 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:21.094102 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:21.094107 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:21.094107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:21.094118 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:21.096478 (+ 2360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:21.096874 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:21.096876 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:21.096877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:21.096973 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:21.097165 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:21.097166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:21.097166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:21.097433 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:21.097561 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:21.097685 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 06:40:21.097817 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":84,"threads_started":1}
I20260928 06:40:21.106177 23391 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:40:21.106688 23391 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:40:21.112205 23391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6a4afc02b974ef99b8ff285bb5487ff: Generated new TSK 0
I20260928 06:40:21.112378 23391 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.211.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab.
Entry for principal kudu/127.18.211.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab.
I20260928 06:40:21.141418 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.65:0
--local_ip_for_outbound_sockets=127.18.211.65
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35449
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:37251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:21.221565 23403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:21.221761 23403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:21.221783 23403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:21.222872 23403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:21.222904 23403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:21.222929 23403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.65
I20260928 06:40:21.224380 23403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:37251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.65
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:21.224555 23403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:21.224736 23403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:21.229501 23408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:21.229594 23403 server_base.cc:1034] running on GCE node
W20260928 06:40:21.229507 23409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:21.229560 23411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:21.229925 23403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:21.230115 23403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:21.231248 23403 hybrid_clock.cc:648] HybridClock initialized: now 1790577621231237 us; error 27 us; skew 500 ppm
Sep 28 06:40:21 dist-test-slave-b28r krb5kdc[23297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577621, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:21.233569 23403 init.cc:377] Logged in from keytab as kudu/127.18.211.65@KRBTEST.COM (short username kudu)
I20260928 06:40:21.233897 23403 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:21.233945 23403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:21.234036 23403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:21.235034 23403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-0/data/instance:
uuid: "654300da6965428ca69bea4f9bf3fdff"
format_stamp: "Formatted at 2026-09-28 06:40:21 on dist-test-slave-b28r"
server_key: "d67d5319fccb27525368aded6d1ad5f9"
server_key_iv: "b55341901ada67746d84ad63ab964691"
server_key_version: "encryptionkey@0"
I20260928 06:40:21.235416 23403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance:
uuid: "654300da6965428ca69bea4f9bf3fdff"
format_stamp: "Formatted at 2026-09-28 06:40:21 on dist-test-slave-b28r"
server_key: "d67d5319fccb27525368aded6d1ad5f9"
server_key_iv: "b55341901ada67746d84ad63ab964691"
server_key_version: "encryptionkey@0"
I20260928 06:40:21.236681 23403 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:21.237370 23417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:21.237512 23403 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:40:21.237596 23403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-0/wal
uuid: "654300da6965428ca69bea4f9bf3fdff"
format_stamp: "Formatted at 2026-09-28 06:40:21 on dist-test-slave-b28r"
server_key: "d67d5319fccb27525368aded6d1ad5f9"
server_key_iv: "b55341901ada67746d84ad63ab964691"
server_key_version: "encryptionkey@0"
I20260928 06:40:21.237655 23403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:21.256815 23403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:21.258800 23403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:21.258934 23403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:21.259203 23403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:21.259531 23403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:21.259567 23403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:21.259595 23403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:21.259619 23403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:21.265168 23403 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.65:34141
I20260928 06:40:21.265211 23530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.65:34141 every 8 connection(s)
I20260928 06:40:21.265470 23403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
I20260928 06:40:21.265990 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 23403
I20260928 06:40:21.266101 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance
I20260928 06:40:21.266299 19277 external_mini_cluster.cc:1428] Setting key fc577933d6e10d78794287c74730ffd3
Sep 28 06:40:21 dist-test-slave-b28r krb5kdc[23297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577621, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.211.66@KRBTEST.COM; defaulting to no policy
I20260928 06:40:21.274887 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:21.266166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:46231 (local address 127.18.211.126:35449)
0928 06:40:21.266221 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:21.266224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:21.266395 (+ 171us) server_negotiation.cc:407] Connection header received
0928 06:40:21.267055 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:21.267055 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:21.267069 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:21.267145 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:21.268059 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.268633 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.269711 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.269999 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.270062 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:21.272140 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:21.272141 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:21.272142 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:21.272152 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:21.273599 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:21.273983 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:21.273984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:21.273985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:21.274048 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:21.274251 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:21.274252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:21.274252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:21.274384 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:21.274454 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:21.274668 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 06:40:21.274749 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:40:21.274952 23533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:21.266111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35449 (local address 127.18.211.65:46231)
0928 06:40:21.266269 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:21.266315 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:21.266946 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:21.267209 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:21.267213 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:21.267463 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:21.267969 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:21.267975 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.268730 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.268731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:21.269633 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:21.269636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.270073 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.270074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:21.270204 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:21.270680 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:21.270689 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:21.272051 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:21.273708 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:21.273714 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:21.273716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:21.273916 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:21.274111 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:21.274111 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:21.274111 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:21.274197 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:21.274442 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:21.274444 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:21.274550 (+ 106us) client_negotiation.cc:769] Sending connection context
0928 06:40:21.274625 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 06:40:21.274756 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260928 06:40:21.275439 23531 heartbeater.cc:347] Connected to a master server at 127.18.211.126:35449
I20260928 06:40:21.275532 23531 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:21.275673 23531 heartbeater.cc:511] Master 127.18.211.126:35449 requested a full tablet report, sending...
I20260928 06:40:21.276242 23341 ts_manager.cc:194] Registered new tserver with Master: 654300da6965428ca69bea4f9bf3fdff (127.18.211.65:34141)
I20260928 06:40:21.276852 23341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.65@KRBTEST.COM'} at 127.18.211.65:46231
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab.
Entry for principal kudu/127.18.211.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab.
I20260928 06:40:21.289776 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.66:0
--local_ip_for_outbound_sockets=127.18.211.66
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35449
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:37251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:21.366895 23536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:21.367101 23536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:21.367134 23536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:21.368268 23536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:21.368314 23536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:21.368350 23536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.66
I20260928 06:40:21.369838 23536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:37251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.66
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:21.370015 23536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:21.370179 23536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:21.374898 23541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:21.374941 23542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:21.374914 23544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:21.375161 23536 server_base.cc:1034] running on GCE node
I20260928 06:40:21.375373 23536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:21.375578 23536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:21.376719 23536 hybrid_clock.cc:648] HybridClock initialized: now 1790577621376707 us; error 31 us; skew 500 ppm
Sep 28 06:40:21 dist-test-slave-b28r krb5kdc[23297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577621, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:21.379151 23536 init.cc:377] Logged in from keytab as kudu/127.18.211.66@KRBTEST.COM (short username kudu)
I20260928 06:40:21.379465 23536 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:21.379510 23536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:21.379601 23536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:21.380570 23536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-1/data/instance:
uuid: "e1ffc3fb56934f48a40fb8f8bcac3b2e"
format_stamp: "Formatted at 2026-09-28 06:40:21 on dist-test-slave-b28r"
server_key: "a6317302aece52846d4f7935036a646e"
server_key_iv: "fe850d08c454ea4998fd3a66e85ff960"
server_key_version: "encryptionkey@0"
I20260928 06:40:21.380851 23536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance:
uuid: "e1ffc3fb56934f48a40fb8f8bcac3b2e"
format_stamp: "Formatted at 2026-09-28 06:40:21 on dist-test-slave-b28r"
server_key: "a6317302aece52846d4f7935036a646e"
server_key_iv: "fe850d08c454ea4998fd3a66e85ff960"
server_key_version: "encryptionkey@0"
I20260928 06:40:21.381991 23536 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:40:21.382655 23550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:21.382779 23536 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:40:21.382861 23536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-1/wal
uuid: "e1ffc3fb56934f48a40fb8f8bcac3b2e"
format_stamp: "Formatted at 2026-09-28 06:40:21 on dist-test-slave-b28r"
server_key: "a6317302aece52846d4f7935036a646e"
server_key_iv: "fe850d08c454ea4998fd3a66e85ff960"
server_key_version: "encryptionkey@0"
I20260928 06:40:21.382925 23536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:21.404398 23536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:21.406107 23536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:21.406234 23536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:21.406451 23536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:21.406745 23536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:21.406780 23536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:21.406800 23536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:21.406811 23536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:21.412498 23536 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.66:43271
I20260928 06:40:21.412549 23663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.66:43271 every 8 connection(s)
I20260928 06:40:21.412782 23536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
I20260928 06:40:21.414214 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 23536
I20260928 06:40:21.414345 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance
I20260928 06:40:21.414535 19277 external_mini_cluster.cc:1428] Setting key 8c1b592884e478ae4765531f29404e44
Sep 28 06:40:21 dist-test-slave-b28r krb5kdc[23297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577621, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.211.67@KRBTEST.COM; defaulting to no policy
I20260928 06:40:21.422475 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:21.413456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:54507 (local address 127.18.211.126:35449)
0928 06:40:21.413539 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:21.413541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:21.413680 (+ 139us) server_negotiation.cc:407] Connection header received
0928 06:40:21.414235 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:21.414237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:21.414249 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:21.414330 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:21.415550 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.416045 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.417158 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.417426 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.417494 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:21.419848 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:21.419850 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:21.419850 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:21.419856 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:21.421049 (+ 1193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:21.421480 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:21.421481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:21.421481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:21.421530 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:21.421789 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:21.421789 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:21.421790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:21.421901 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:21.421962 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:21.422292 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 06:40:21.422380 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 06:40:21.422567 23666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:21.413385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35449 (local address 127.18.211.66:54507)
0928 06:40:21.413558 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:21.413603 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:21.414119 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:21.414454 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:21.414459 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:21.414802 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:21.415458 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:21.415466 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.416123 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.416124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:21.417070 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:21.417073 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.417524 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.417525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:21.417630 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:21.418104 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:21.418113 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:21.419761 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:21.421128 (+ 1367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:21.421135 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:21.421138 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:21.421412 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:21.421586 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:21.421587 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:21.421587 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:21.421715 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:21.421962 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:21.421968 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:21.422140 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 06:40:21.422258 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 06:40:21.422400 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
I20260928 06:40:21.423066 23664 heartbeater.cc:347] Connected to a master server at 127.18.211.126:35449
I20260928 06:40:21.423170 23664 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:21.423296 23664 heartbeater.cc:511] Master 127.18.211.126:35449 requested a full tablet report, sending...
I20260928 06:40:21.423609 23341 ts_manager.cc:194] Registered new tserver with Master: e1ffc3fb56934f48a40fb8f8bcac3b2e (127.18.211.66:43271)
I20260928 06:40:21.423974 23341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.66@KRBTEST.COM'} at 127.18.211.66:54507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab.
Entry for principal kudu/127.18.211.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab.
I20260928 06:40:21.437069 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.67:0
--local_ip_for_outbound_sockets=127.18.211.67
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35449
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:37251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:21.516641 23669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:21.516849 23669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:21.516880 23669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:21.518004 23669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:21.518044 23669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:21.518079 23669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.67
I20260928 06:40:21.519563 23669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:37251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.67
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:21.519784 23669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:21.519982 23669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:21.524735 23674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:21.524760 23675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:21.524762 23677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:21.524796 23669 server_base.cc:1034] running on GCE node
I20260928 06:40:21.525115 23669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:21.525357 23669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:21.526502 23669 hybrid_clock.cc:648] HybridClock initialized: now 1790577621526488 us; error 28 us; skew 500 ppm
Sep 28 06:40:21 dist-test-slave-b28r krb5kdc[23297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577621, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:21.528879 23669 init.cc:377] Logged in from keytab as kudu/127.18.211.67@KRBTEST.COM (short username kudu)
I20260928 06:40:21.529232 23669 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:21.529289 23669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:21.529412 23669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:21.530391 23669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-2/data/instance:
uuid: "77b4305cb18f438680fc12344e37ad49"
format_stamp: "Formatted at 2026-09-28 06:40:21 on dist-test-slave-b28r"
server_key: "c4d3c896893fb1daf3e94b440f6ce983"
server_key_iv: "3647e86395017532e35d48f75cc96b3b"
server_key_version: "encryptionkey@0"
I20260928 06:40:21.530705 23669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance:
uuid: "77b4305cb18f438680fc12344e37ad49"
format_stamp: "Formatted at 2026-09-28 06:40:21 on dist-test-slave-b28r"
server_key: "c4d3c896893fb1daf3e94b440f6ce983"
server_key_iv: "3647e86395017532e35d48f75cc96b3b"
server_key_version: "encryptionkey@0"
I20260928 06:40:21.531984 23669 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:40:21.532681 23683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:21.532838 23669 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:40:21.532917 23669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-2/wal
uuid: "77b4305cb18f438680fc12344e37ad49"
format_stamp: "Formatted at 2026-09-28 06:40:21 on dist-test-slave-b28r"
server_key: "c4d3c896893fb1daf3e94b440f6ce983"
server_key_iv: "3647e86395017532e35d48f75cc96b3b"
server_key_version: "encryptionkey@0"
I20260928 06:40:21.532981 23669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:21.543841 23669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:21.545876 23669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:21.546017 23669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:21.546239 23669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:21.546523 23669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:21.546556 23669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:21.546586 23669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:21.546612 23669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:21.551945 23669 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.67:44917
I20260928 06:40:21.552016 23796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.67:44917 every 8 connection(s)
I20260928 06:40:21.552240 23669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:40:21 dist-test-slave-b28r krb5kdc[23297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577621, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:21.560562 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:21.552917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:38671 (local address 127.18.211.126:35449)
0928 06:40:21.552969 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:21.552971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:21.553163 (+ 192us) server_negotiation.cc:407] Connection header received
0928 06:40:21.553640 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:21.553641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:21.553650 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:21.553693 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:21.554545 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.555045 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.556033 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.556219 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.556275 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:21.558168 (+ 1893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:21.558169 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:21.558170 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:21.558176 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:21.559283 (+ 1107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:21.559615 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:21.559616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:21.559616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:21.559665 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:21.559874 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:21.559876 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:21.559876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:21.560018 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:21.560093 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:21.560420 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 06:40:21.560491 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 06:40:21.560782 23799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:21.552879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35449 (local address 127.18.211.67:38671)
0928 06:40:21.553044 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:21.553085 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:21.553539 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:21.553755 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:21.553758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:21.554002 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:21.554469 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:21.554474 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.555155 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.555156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:21.555957 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:21.555959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.556287 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.556288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:21.556384 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:21.556827 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:21.556836 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:21.558089 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:21.559347 (+ 1258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:21.559352 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:21.559354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:21.559548 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:21.559709 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:21.559710 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:21.559710 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:21.559797 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:21.560108 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:21.560114 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:21.560272 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 06:40:21.560390 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 06:40:21.560527 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":82,"threads_started":1}
I20260928 06:40:21.561305 23797 heartbeater.cc:347] Connected to a master server at 127.18.211.126:35449
I20260928 06:40:21.561401 23797 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:21.561393 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 23669
I20260928 06:40:21.561499 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance
I20260928 06:40:21.561544 23797 heartbeater.cc:511] Master 127.18.211.126:35449 requested a full tablet report, sending...
I20260928 06:40:21.561677 19277 external_mini_cluster.cc:1428] Setting key eef9e2bca3159bf0d9c3616e2546c3a9
I20260928 06:40:21.561918 23341 ts_manager.cc:194] Registered new tserver with Master: 77b4305cb18f438680fc12344e37ad49 (127.18.211.67:44917)
I20260928 06:40:21.562346 23341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.67@KRBTEST.COM'} at 127.18.211.67:38671
I20260928 06:40:21.562436 19277 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:40:21.569303 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:21.563816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40680 (local address 127.18.211.126:35449)
0928 06:40:21.563880 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:21.563881 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:21.563967 (+ 86us) server_negotiation.cc:407] Connection header received
0928 06:40:21.564024 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:21.564025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:21.564033 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:21.564076 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:21.564614 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.565116 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.565811 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.565996 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.566048 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:21.567103 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:21.567104 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:21.567105 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:21.567111 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:21.568277 (+ 1166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:21.568600 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:21.568601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:21.568601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:21.568651 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:21.568826 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:21.568827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:21.568827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:21.568968 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:21.569051 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:21.569154 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 06:40:21.569222 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:40:21.570685 23341 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:40:21.572345 23341 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:40:21.578181 23805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:21.574315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:44917 (local address 127.0.0.1:46666)
0928 06:40:21.574574 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:21.574617 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:21.574724 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:21.574991 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:21.574993 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:21.574995 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:21.575057 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:21.575059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.576055 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.576055 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:21.577105 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:21.577107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.577945 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.577946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:21.578000 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:21.578008 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 06:40:21.578061 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 06:40:21.578110 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":86,"threads_started":1}
I20260928 06:40:21.578209 23807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:21.574393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46666 (local address 127.18.211.67:44917)
0928 06:40:21.574574 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:21.574580 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:21.574693 (+ 113us) server_negotiation.cc:407] Connection header received
0928 06:40:21.574828 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:21.574831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:21.574848 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:21.574899 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:21.575223 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.575970 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.577198 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.577830 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.577863 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:21.577899 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:21.578093 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 06:40:21.578129 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":92,"threads_started":1}
I20260928 06:40:21.578580 23808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:21.574334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:34141 (local address 127.0.0.1:56588)
0928 06:40:21.574646 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:21.574655 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:21.574721 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:21.575337 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:21.575337 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:21.575339 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:21.575374 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:21.575375 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.576534 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.576535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:21.577438 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:21.577440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.578369 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.578369 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:21.578427 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:21.578430 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:40:21.578472 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 06:40:21.578511 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":51,"threads_started":1}
I20260928 06:40:21.578778 23806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:21.574519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:43271 (local address 127.0.0.1:51128)
0928 06:40:21.574694 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:21.574703 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:21.574741 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:21.575069 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:21.575069 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:21.575072 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:21.575150 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:21.575151 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.576500 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.576501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:21.577546 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:21.577548 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.578599 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.578600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:21.578656 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:21.578659 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:40:21.578688 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 06:40:21.578720 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":86,"threads_started":1}
I20260928 06:40:21.578789 23809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:21.574506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56588 (local address 127.18.211.65:34141)
0928 06:40:21.574880 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:21.574885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:21.574895 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:40:21.574934 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:21.574936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:21.574951 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:21.575027 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:21.575444 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.576464 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.577508 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.578209 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.578277 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:21.578335 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:21.578673 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 06:40:21.578713 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":100,"threads_started":1}
I20260928 06:40:21.578922 23731 tablet_service.cc:1467] Processing CreateTablet for tablet ea57adbe401d490cbda47a309f6a3c0d (DEFAULT_TABLE table=test-table [id=c2cdcf5a8a274076ac6ea4c3fedee671]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:21.579062 23810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:21.574752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51128 (local address 127.18.211.66:43271)
0928 06:40:21.574912 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:21.574916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:21.574924 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:40:21.574958 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:21.574959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:21.574975 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:21.575221 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:21.575302 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.576398 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.577616 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.578520 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.578903 (+ 383us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:21.578956 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:21.578976 (+ 20us) server_negotiation.cc:299] Negotiation successful
0928 06:40:21.579006 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":88,"threads_started":1}
I20260928 06:40:21.579463 23731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea57adbe401d490cbda47a309f6a3c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:21.579715 23465 tablet_service.cc:1467] Processing CreateTablet for tablet ea57adbe401d490cbda47a309f6a3c0d (DEFAULT_TABLE table=test-table [id=c2cdcf5a8a274076ac6ea4c3fedee671]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:21.579715 23598 tablet_service.cc:1467] Processing CreateTablet for tablet ea57adbe401d490cbda47a309f6a3c0d (DEFAULT_TABLE table=test-table [id=c2cdcf5a8a274076ac6ea4c3fedee671]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:21.580036 23465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea57adbe401d490cbda47a309f6a3c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:21.580036 23598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea57adbe401d490cbda47a309f6a3c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:21.582203 23811 tablet_bootstrap.cc:492] T ea57adbe401d490cbda47a309f6a3c0d P 77b4305cb18f438680fc12344e37ad49: Bootstrap starting.
I20260928 06:40:21.582782 23811 tablet_bootstrap.cc:654] T ea57adbe401d490cbda47a309f6a3c0d P 77b4305cb18f438680fc12344e37ad49: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:21.583133 23813 tablet_bootstrap.cc:492] T ea57adbe401d490cbda47a309f6a3c0d P e1ffc3fb56934f48a40fb8f8bcac3b2e: Bootstrap starting.
I20260928 06:40:21.583364 23811 log.cc:824] T ea57adbe401d490cbda47a309f6a3c0d P 77b4305cb18f438680fc12344e37ad49: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:21.583772 23813 tablet_bootstrap.cc:654] T ea57adbe401d490cbda47a309f6a3c0d P e1ffc3fb56934f48a40fb8f8bcac3b2e: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:21.584101 23811 tablet_bootstrap.cc:492] T ea57adbe401d490cbda47a309f6a3c0d P 77b4305cb18f438680fc12344e37ad49: No bootstrap required, opened a new log
I20260928 06:40:21.584174 23811 ts_tablet_manager.cc:1397] T ea57adbe401d490cbda47a309f6a3c0d P 77b4305cb18f438680fc12344e37ad49: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:40:21.584359 23813 log.cc:824] T ea57adbe401d490cbda47a309f6a3c0d P e1ffc3fb56934f48a40fb8f8bcac3b2e: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:21.584693 23812 tablet_bootstrap.cc:492] T ea57adbe401d490cbda47a309f6a3c0d P 654300da6965428ca69bea4f9bf3fdff: Bootstrap starting.
I20260928 06:40:21.584991 23813 tablet_bootstrap.cc:492] T ea57adbe401d490cbda47a309f6a3c0d P e1ffc3fb56934f48a40fb8f8bcac3b2e: No bootstrap required, opened a new log
I20260928 06:40:21.585048 23813 ts_tablet_manager.cc:1397] T ea57adbe401d490cbda47a309f6a3c0d P e1ffc3fb56934f48a40fb8f8bcac3b2e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:40:21.585291 23812 tablet_bootstrap.cc:654] T ea57adbe401d490cbda47a309f6a3c0d P 654300da6965428ca69bea4f9bf3fdff: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:21.585896 23812 log.cc:824] T ea57adbe401d490cbda47a309f6a3c0d P 654300da6965428ca69bea4f9bf3fdff: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:21.586093 23811 raft_consensus.cc:348] T ea57adbe401d490cbda47a309f6a3c0d P 77b4305cb18f438680fc12344e37ad49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b4305cb18f438680fc12344e37ad49" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 44917 } } peers { permanent_uuid: "654300da6965428ca69bea4f9bf3fdff" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 34141 } } peers { permanent_uuid: "e1ffc3fb56934f48a40fb8f8bcac3b2e" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 43271 } }
I20260928 06:40:21.586247 23811 raft_consensus.cc:374] T ea57adbe401d490cbda47a309f6a3c0d P 77b4305cb18f438680fc12344e37ad49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:21.586282 23811 raft_consensus.cc:729] T ea57adbe401d490cbda47a309f6a3c0d P 77b4305cb18f438680fc12344e37ad49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77b4305cb18f438680fc12344e37ad49, State: Initialized, Role: FOLLOWER
I20260928 06:40:21.586390 23811 consensus_queue.cc:260] T ea57adbe401d490cbda47a309f6a3c0d P 77b4305cb18f438680fc12344e37ad49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b4305cb18f438680fc12344e37ad49" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 44917 } } peers { permanent_uuid: "654300da6965428ca69bea4f9bf3fdff" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 34141 } } peers { permanent_uuid: "e1ffc3fb56934f48a40fb8f8bcac3b2e" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 43271 } }
I20260928 06:40:21.586661 23811 ts_tablet_manager.cc:1428] T ea57adbe401d490cbda47a309f6a3c0d P 77b4305cb18f438680fc12344e37ad49: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:40:21.586774 23812 tablet_bootstrap.cc:492] T ea57adbe401d490cbda47a309f6a3c0d P 654300da6965428ca69bea4f9bf3fdff: No bootstrap required, opened a new log
I20260928 06:40:21.586802 23797 heartbeater.cc:503] Master 127.18.211.126:35449 was elected leader, sending a full tablet report...
I20260928 06:40:21.586833 23812 ts_tablet_manager.cc:1397] T ea57adbe401d490cbda47a309f6a3c0d P 654300da6965428ca69bea4f9bf3fdff: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:40:21.587116 23813 raft_consensus.cc:348] T ea57adbe401d490cbda47a309f6a3c0d P e1ffc3fb56934f48a40fb8f8bcac3b2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b4305cb18f438680fc12344e37ad49" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 44917 } } peers { permanent_uuid: "654300da6965428ca69bea4f9bf3fdff" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 34141 } } peers { permanent_uuid: "e1ffc3fb56934f48a40fb8f8bcac3b2e" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 43271 } }
I20260928 06:40:21.587267 23813 raft_consensus.cc:374] T ea57adbe401d490cbda47a309f6a3c0d P e1ffc3fb56934f48a40fb8f8bcac3b2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:21.587308 23813 raft_consensus.cc:729] T ea57adbe401d490cbda47a309f6a3c0d P e1ffc3fb56934f48a40fb8f8bcac3b2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1ffc3fb56934f48a40fb8f8bcac3b2e, State: Initialized, Role: FOLLOWER
I20260928 06:40:21.587476 23813 consensus_queue.cc:260] T ea57adbe401d490cbda47a309f6a3c0d P e1ffc3fb56934f48a40fb8f8bcac3b2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b4305cb18f438680fc12344e37ad49" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 44917 } } peers { permanent_uuid: "654300da6965428ca69bea4f9bf3fdff" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 34141 } } peers { permanent_uuid: "e1ffc3fb56934f48a40fb8f8bcac3b2e" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 43271 } }
I20260928 06:40:21.587734 23813 ts_tablet_manager.cc:1428] T ea57adbe401d490cbda47a309f6a3c0d P e1ffc3fb56934f48a40fb8f8bcac3b2e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:40:21.587733 23664 heartbeater.cc:503] Master 127.18.211.126:35449 was elected leader, sending a full tablet report...
I20260928 06:40:21.587836 23818 raft_consensus.cc:482] T ea57adbe401d490cbda47a309f6a3c0d P e1ffc3fb56934f48a40fb8f8bcac3b2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:40:21.587929 23818 raft_consensus.cc:504] T ea57adbe401d490cbda47a309f6a3c0d P e1ffc3fb56934f48a40fb8f8bcac3b2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b4305cb18f438680fc12344e37ad49" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 44917 } } peers { permanent_uuid: "654300da6965428ca69bea4f9bf3fdff" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 34141 } } peers { permanent_uuid: "e1ffc3fb56934f48a40fb8f8bcac3b2e" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 43271 } }
I20260928 06:40:21.588807 23818 leader_election.cc:288] T ea57adbe401d490cbda47a309f6a3c0d P e1ffc3fb56934f48a40fb8f8bcac3b2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77b4305cb18f438680fc12344e37ad49 (127.18.211.67:44917), 654300da6965428ca69bea4f9bf3fdff (127.18.211.65:34141)
I20260928 06:40:21.588848 23812 raft_consensus.cc:348] T ea57adbe401d490cbda47a309f6a3c0d P 654300da6965428ca69bea4f9bf3fdff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b4305cb18f438680fc12344e37ad49" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 44917 } } peers { permanent_uuid: "654300da6965428ca69bea4f9bf3fdff" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 34141 } } peers { permanent_uuid: "e1ffc3fb56934f48a40fb8f8bcac3b2e" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 43271 } }
I20260928 06:40:21.589015 23812 raft_consensus.cc:374] T ea57adbe401d490cbda47a309f6a3c0d P 654300da6965428ca69bea4f9bf3fdff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:21.589062 23812 raft_consensus.cc:729] T ea57adbe401d490cbda47a309f6a3c0d P 654300da6965428ca69bea4f9bf3fdff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 654300da6965428ca69bea4f9bf3fdff, State: Initialized, Role: FOLLOWER
I20260928 06:40:21.589166 23812 consensus_queue.cc:260] T ea57adbe401d490cbda47a309f6a3c0d P 654300da6965428ca69bea4f9bf3fdff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b4305cb18f438680fc12344e37ad49" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 44917 } } peers { permanent_uuid: "654300da6965428ca69bea4f9bf3fdff" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 34141 } } peers { permanent_uuid: "e1ffc3fb56934f48a40fb8f8bcac3b2e" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 43271 } }
I20260928 06:40:21.589443 23812 ts_tablet_manager.cc:1428] T ea57adbe401d490cbda47a309f6a3c0d P 654300da6965428ca69bea4f9bf3fdff: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:40:21.589476 23531 heartbeater.cc:503] Master 127.18.211.126:35449 was elected leader, sending a full tablet report...
I20260928 06:40:21.591774 23666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:21.588302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:44917 (local address 127.18.211.66:56837)
0928 06:40:21.588343 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:21.588360 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:21.588393 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:21.588532 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:21.588532 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:21.588534 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:21.588597 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:21.588599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.590013 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.590014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:21.591106 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:21.591109 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.591591 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.591591 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:21.591664 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:21.591667 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:40:21.591702 (+ 35us) client_negotiation.cc:240] Negotiation successful
0928 06:40:21.591731 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 06:40:21.591809 23807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:21.588337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:56837 (local address 127.18.211.67:44917)
0928 06:40:21.588376 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:21.588378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:21.588388 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:40:21.588440 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:21.588441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:21.588449 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:21.588492 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:21.588700 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.589389 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.591175 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.591524 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.591588 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:21.591595 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:21.591718 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 06:40:21.591750 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:40:21.592041 23751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea57adbe401d490cbda47a309f6a3c0d" candidate_uuid: "e1ffc3fb56934f48a40fb8f8bcac3b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b4305cb18f438680fc12344e37ad49" is_pre_election: true
I20260928 06:40:21.592145 23751 raft_consensus.cc:2454] T ea57adbe401d490cbda47a309f6a3c0d P 77b4305cb18f438680fc12344e37ad49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1ffc3fb56934f48a40fb8f8bcac3b2e in term 0.
I20260928 06:40:21.592363 23809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:21.588754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:46409 (local address 127.18.211.65:34141)
0928 06:40:21.588790 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:21.588792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:21.589645 (+ 853us) server_negotiation.cc:407] Connection header received
0928 06:40:21.589669 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:21.589669 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:21.589679 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:21.589746 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:21.589962 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.590553 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.591590 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.592046 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.592101 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:21.592107 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:21.592275 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 06:40:21.592309 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:40:21.592342 23554 leader_election.cc:302] T ea57adbe401d490cbda47a309f6a3c0d P e1ffc3fb56934f48a40fb8f8bcac3b2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77b4305cb18f438680fc12344e37ad49, e1ffc3fb56934f48a40fb8f8bcac3b2e; no voters:
I20260928 06:40:21.592389 23820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:21.588551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:34141 (local address 127.18.211.66:46409)
0928 06:40:21.589556 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:21.589566 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:21.589625 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:21.589800 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:21.589801 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:21.589802 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:21.589870 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:21.589871 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.590617 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.590618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:21.591522 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:21.591524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.592132 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.592134 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:21.592198 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:21.592202 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:40:21.592267 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 06:40:21.592302 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":988,"thread_start_us":101,"threads_started":1}
I20260928 06:40:21.592480 23818 raft_consensus.cc:2781] T ea57adbe401d490cbda47a309f6a3c0d P e1ffc3fb56934f48a40fb8f8bcac3b2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:40:21.592538 23818 raft_consensus.cc:482] T ea57adbe401d490cbda47a309f6a3c0d P e1ffc3fb56934f48a40fb8f8bcac3b2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:40:21.592558 23818 raft_consensus.cc:3037] T ea57adbe401d490cbda47a309f6a3c0d P e1ffc3fb56934f48a40fb8f8bcac3b2e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:21.592657 23485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea57adbe401d490cbda47a309f6a3c0d" candidate_uuid: "e1ffc3fb56934f48a40fb8f8bcac3b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "654300da6965428ca69bea4f9bf3fdff" is_pre_election: true
I20260928 06:40:21.592773 23485 raft_consensus.cc:2454] T ea57adbe401d490cbda47a309f6a3c0d P 654300da6965428ca69bea4f9bf3fdff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1ffc3fb56934f48a40fb8f8bcac3b2e in term 0.
I20260928 06:40:21.593175 23818 raft_consensus.cc:504] T ea57adbe401d490cbda47a309f6a3c0d P e1ffc3fb56934f48a40fb8f8bcac3b2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b4305cb18f438680fc12344e37ad49" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 44917 } } peers { permanent_uuid: "654300da6965428ca69bea4f9bf3fdff" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 34141 } } peers { permanent_uuid: "e1ffc3fb56934f48a40fb8f8bcac3b2e" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 43271 } }
I20260928 06:40:21.593294 23818 leader_election.cc:288] T ea57adbe401d490cbda47a309f6a3c0d P e1ffc3fb56934f48a40fb8f8bcac3b2e [CANDIDATE]: Term 1 election: Requested vote from peers 77b4305cb18f438680fc12344e37ad49 (127.18.211.67:44917), 654300da6965428ca69bea4f9bf3fdff (127.18.211.65:34141)
I20260928 06:40:21.593430 23485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea57adbe401d490cbda47a309f6a3c0d" candidate_uuid: "e1ffc3fb56934f48a40fb8f8bcac3b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "654300da6965428ca69bea4f9bf3fdff"
I20260928 06:40:21.593461 23751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea57adbe401d490cbda47a309f6a3c0d" candidate_uuid: "e1ffc3fb56934f48a40fb8f8bcac3b2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77b4305cb18f438680fc12344e37ad49"
I20260928 06:40:21.593499 23485 raft_consensus.cc:3037] T ea57adbe401d490cbda47a309f6a3c0d P 654300da6965428ca69bea4f9bf3fdff [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:21.593513 23751 raft_consensus.cc:3037] T ea57adbe401d490cbda47a309f6a3c0d P 77b4305cb18f438680fc12344e37ad49 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:21.594089 23485 raft_consensus.cc:2454] T ea57adbe401d490cbda47a309f6a3c0d P 654300da6965428ca69bea4f9bf3fdff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1ffc3fb56934f48a40fb8f8bcac3b2e in term 1.
I20260928 06:40:21.594097 23751 raft_consensus.cc:2454] T ea57adbe401d490cbda47a309f6a3c0d P 77b4305cb18f438680fc12344e37ad49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1ffc3fb56934f48a40fb8f8bcac3b2e in term 1.
I20260928 06:40:21.594259 23553 leader_election.cc:302] T ea57adbe401d490cbda47a309f6a3c0d P e1ffc3fb56934f48a40fb8f8bcac3b2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 654300da6965428ca69bea4f9bf3fdff, e1ffc3fb56934f48a40fb8f8bcac3b2e; no voters:
I20260928 06:40:21.594355 23818 raft_consensus.cc:2781] T ea57adbe401d490cbda47a309f6a3c0d P e1ffc3fb56934f48a40fb8f8bcac3b2e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:21.594417 23818 raft_consensus.cc:686] T ea57adbe401d490cbda47a309f6a3c0d P e1ffc3fb56934f48a40fb8f8bcac3b2e [term 1 LEADER]: Becoming Leader. State: Replica: e1ffc3fb56934f48a40fb8f8bcac3b2e, State: Running, Role: LEADER
I20260928 06:40:21.594481 23818 consensus_queue.cc:237] T ea57adbe401d490cbda47a309f6a3c0d P e1ffc3fb56934f48a40fb8f8bcac3b2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b4305cb18f438680fc12344e37ad49" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 44917 } } peers { permanent_uuid: "654300da6965428ca69bea4f9bf3fdff" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 34141 } } peers { permanent_uuid: "e1ffc3fb56934f48a40fb8f8bcac3b2e" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 43271 } }
I20260928 06:40:21.596014 23340 catalog_manager.cc:5495] T ea57adbe401d490cbda47a309f6a3c0d P e1ffc3fb56934f48a40fb8f8bcac3b2e reported cstate change: term changed from 0 to 1, leader changed from <none> to e1ffc3fb56934f48a40fb8f8bcac3b2e (127.18.211.66). New cstate: current_term: 1 leader_uuid: "e1ffc3fb56934f48a40fb8f8bcac3b2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77b4305cb18f438680fc12344e37ad49" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 44917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "654300da6965428ca69bea4f9bf3fdff" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 34141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1ffc3fb56934f48a40fb8f8bcac3b2e" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 43271 } health_report { overall_health: HEALTHY } } }
I20260928 06:40:21.606372 23810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:21.604198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51140 (local address 127.18.211.66:43271)
0928 06:40:21.604255 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:21.604260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:21.604268 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:40:21.604320 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:21.604321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:21.604340 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:21.604400 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:40:21.604525 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.604976 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.605680 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:21.605876 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:21.605927 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:21.606073 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:40:21.606168 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:40:21.606241 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:21.606277 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 06:40:21.606297 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 06:40:21.607993 23751 raft_consensus.cc:1261] T ea57adbe401d490cbda47a309f6a3c0d P 77b4305cb18f438680fc12344e37ad49 [term 1 FOLLOWER]: Refusing update from remote peer e1ffc3fb56934f48a40fb8f8bcac3b2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:40:21.607996 23485 raft_consensus.cc:1261] T ea57adbe401d490cbda47a309f6a3c0d P 654300da6965428ca69bea4f9bf3fdff [term 1 FOLLOWER]: Refusing update from remote peer e1ffc3fb56934f48a40fb8f8bcac3b2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:40:21.608314 23818 consensus_queue.cc:1035] T ea57adbe401d490cbda47a309f6a3c0d P e1ffc3fb56934f48a40fb8f8bcac3b2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "654300da6965428ca69bea4f9bf3fdff" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 34141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:40:21.608448 23819 consensus_queue.cc:1035] T ea57adbe401d490cbda47a309f6a3c0d P e1ffc3fb56934f48a40fb8f8bcac3b2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "77b4305cb18f438680fc12344e37ad49" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 44917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:40:21.611042 23826 mvcc.cc:205] Tried to move back new op lower bound from 7334205938103971840 to 7334205938051551232. Current Snapshot: MvccSnapshot[applied={T|T < 7334205938103971840}]
I20260928 06:40:21.612499 23339 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:40:21.612574 23339 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:40:21.613257 23339 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6a4afc02b974ef99b8ff285bb5487ff: Sending DeleteTablet for 3 replicas of tablet ea57adbe401d490cbda47a309f6a3c0d
I20260928 06:40:21.613569 23731 tablet_service.cc:1514] Processing DeleteTablet for tablet ea57adbe401d490cbda47a309f6a3c0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:21 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:46666
I20260928 06:40:21.613571 23465 tablet_service.cc:1514] Processing DeleteTablet for tablet ea57adbe401d490cbda47a309f6a3c0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:21 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:56588
I20260928 06:40:21.613799 23834 tablet_replica.cc:330] T ea57adbe401d490cbda47a309f6a3c0d P 654300da6965428ca69bea4f9bf3fdff: stopping tablet replica
I20260928 06:40:21.613829 23598 tablet_service.cc:1514] Processing DeleteTablet for tablet ea57adbe401d490cbda47a309f6a3c0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:21 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:51128
I20260928 06:40:21.613902 23834 raft_consensus.cc:2229] T ea57adbe401d490cbda47a309f6a3c0d P 654300da6965428ca69bea4f9bf3fdff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:40:21.614004 23834 raft_consensus.cc:2258] T ea57adbe401d490cbda47a309f6a3c0d P 654300da6965428ca69bea4f9bf3fdff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:21.614032 23836 tablet_replica.cc:330] T ea57adbe401d490cbda47a309f6a3c0d P e1ffc3fb56934f48a40fb8f8bcac3b2e: stopping tablet replica
I20260928 06:40:21.614157 23835 tablet_replica.cc:330] T ea57adbe401d490cbda47a309f6a3c0d P 77b4305cb18f438680fc12344e37ad49: stopping tablet replica
I20260928 06:40:21.614233 23836 raft_consensus.cc:2229] T ea57adbe401d490cbda47a309f6a3c0d P e1ffc3fb56934f48a40fb8f8bcac3b2e [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:40:21.614251 23835 raft_consensus.cc:2229] T ea57adbe401d490cbda47a309f6a3c0d P 77b4305cb18f438680fc12344e37ad49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:40:21.614339 23835 raft_consensus.cc:2258] T ea57adbe401d490cbda47a309f6a3c0d P 77b4305cb18f438680fc12344e37ad49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:21.614332 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 23403
I20260928 06:40:21.614899 23836 raft_consensus.cc:2258] T ea57adbe401d490cbda47a309f6a3c0d P e1ffc3fb56934f48a40fb8f8bcac3b2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:21.615481 23836 ts_tablet_manager.cc:1905] T ea57adbe401d490cbda47a309f6a3c0d P e1ffc3fb56934f48a40fb8f8bcac3b2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:21.617054 23836 ts_tablet_manager.cc:1918] T ea57adbe401d490cbda47a309f6a3c0d P e1ffc3fb56934f48a40fb8f8bcac3b2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:40:21.617170 23836 log.cc:1196] T ea57adbe401d490cbda47a309f6a3c0d P e1ffc3fb56934f48a40fb8f8bcac3b2e: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-1/wal/wals/ea57adbe401d490cbda47a309f6a3c0d
I20260928 06:40:21.617501 23836 ts_tablet_manager.cc:1939] T ea57adbe401d490cbda47a309f6a3c0d P e1ffc3fb56934f48a40fb8f8bcac3b2e: Deleting consensus metadata
I20260928 06:40:21.617897 23327 catalog_manager.cc:4841] TS e1ffc3fb56934f48a40fb8f8bcac3b2e (127.18.211.66:43271): tablet ea57adbe401d490cbda47a309f6a3c0d (table test-table [id=c2cdcf5a8a274076ac6ea4c3fedee671]) successfully deleted
I20260928 06:40:21.619506 23835 ts_tablet_manager.cc:1905] T ea57adbe401d490cbda47a309f6a3c0d P 77b4305cb18f438680fc12344e37ad49: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 06:40:21.620178 23327 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.211.65:34141 (error 108)
W20260928 06:40:21.620594 23806 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:21.620492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:34141 (local address 127.0.0.1:56590)
0928 06:40:21.620532 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:21.620561 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.65:34141: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 06:40:21.620738 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 23536
W20260928 06:40:21.620780 23327 catalog_manager.cc:4584] TS 654300da6965428ca69bea4f9bf3fdff (127.18.211.65:34141): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ea57adbe401d490cbda47a309f6a3c0d: Network error: Client connection negotiation failed: client connection to 127.18.211.65:34141: connect: Connection reset by peer (error 104)
I20260928 06:40:21.621445 23835 ts_tablet_manager.cc:1918] T ea57adbe401d490cbda47a309f6a3c0d P 77b4305cb18f438680fc12344e37ad49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:40:21.621501 23835 log.cc:1196] T ea57adbe401d490cbda47a309f6a3c0d P 77b4305cb18f438680fc12344e37ad49: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790577602522428-19277-0/minicluster-data/ts-2/wal/wals/ea57adbe401d490cbda47a309f6a3c0d
I20260928 06:40:21.621716 23835 ts_tablet_manager.cc:1939] T ea57adbe401d490cbda47a309f6a3c0d P 77b4305cb18f438680fc12344e37ad49: Deleting consensus metadata
I20260928 06:40:21.622226 23328 catalog_manager.cc:4841] TS 77b4305cb18f438680fc12344e37ad49 (127.18.211.67:44917): tablet ea57adbe401d490cbda47a309f6a3c0d (table test-table [id=c2cdcf5a8a274076ac6ea4c3fedee671]) successfully deleted
I20260928 06:40:21.625435 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 23669
I20260928 06:40:21.629437 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 23310
2026-09-28T06:40:21Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1703 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T06:40:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:40:21Z Disabled control of system clock
I20260928 06:40:21.656471 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:35053
--webserver_interface=127.18.211.126
--webserver_port=0
--builtin_ntp_servers=127.18.211.84:35905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:35053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 06:40:21.735057 23841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:21.735291 23841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:21.735311 23841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:21.736387 23841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:40:21.736418 23841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:21.736429 23841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:21.736440 23841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:40:21.736450 23841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:40:21.737699 23841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:35905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:35053
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:35053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:21.737891 23841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:21.738121 23841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:21.742573 23849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:21.742563 23846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:21.742563 23847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:21.743223 23841 server_base.cc:1034] running on GCE node
I20260928 06:40:21.743463 23841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:21.743716 23841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:21.744853 23841 hybrid_clock.cc:648] HybridClock initialized: now 1790577621744840 us; error 28 us; skew 500 ppm
I20260928 06:40:21.746003 23841 webserver.cc:460] Webserver started at http://127.18.211.126:45291/ using document root <none> and password file <none>
I20260928 06:40:21.746215 23841 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:21.746263 23841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:21.746394 23841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:21.747455 23841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "bffec50127a84b4eb1d49d310e94998a"
format_stamp: "Formatted at 2026-09-28 06:40:21 on dist-test-slave-b28r"
server_key: "c58e3e10784a78e5a0f913e0be841605"
server_key_iv: "2a97885b4db2da4dc66ffe87561ff5a3"
server_key_version: "encryptionkey@0"
I20260928 06:40:21.747776 23841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "bffec50127a84b4eb1d49d310e94998a"
format_stamp: "Formatted at 2026-09-28 06:40:21 on dist-test-slave-b28r"
server_key: "c58e3e10784a78e5a0f913e0be841605"
server_key_iv: "2a97885b4db2da4dc66ffe87561ff5a3"
server_key_version: "encryptionkey@0"
I20260928 06:40:21.748966 23841 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 06:40:21.749742 23855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:21.749913 23841 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:40:21.749993 23841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "bffec50127a84b4eb1d49d310e94998a"
format_stamp: "Formatted at 2026-09-28 06:40:21 on dist-test-slave-b28r"
server_key: "c58e3e10784a78e5a0f913e0be841605"
server_key_iv: "2a97885b4db2da4dc66ffe87561ff5a3"
server_key_version: "encryptionkey@0"
I20260928 06:40:21.750052 23841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:21.753242 23841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:21.753482 23841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:21.753597 23841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:21.757184 23841 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:35053
I20260928 06:40:21.757221 23907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:35053 every 8 connection(s)
I20260928 06:40:21.757511 23841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:40:21.758126 23908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:21.760442 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 23841
I20260928 06:40:21.760535 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:40:21.760686 19277 external_mini_cluster.cc:1428] Setting key efa4143a526052cf8ad339ca94ae3c2f
I20260928 06:40:21.761003 23908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bffec50127a84b4eb1d49d310e94998a: Bootstrap starting.
I20260928 06:40:21.761689 23908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bffec50127a84b4eb1d49d310e94998a: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:21.762022 23908 log.cc:824] T 00000000000000000000000000000000 P bffec50127a84b4eb1d49d310e94998a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:21.763111 23908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bffec50127a84b4eb1d49d310e94998a: No bootstrap required, opened a new log
I20260928 06:40:21.763437 23911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:21.761198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33808 (local address 127.18.211.126:35053)
0928 06:40:21.761428 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:21.761437 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:21.761471 (+ 34us) server_negotiation.cc:407] Connection header received
0928 06:40:21.762057 (+ 586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:21.762067 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:21.762119 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:21.762292 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:21.762911 (+ 619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:21.762915 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:21.762915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:40:21.762920 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:21.762971 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:40:21.762985 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:21.763113 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:21.763162 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 06:40:21.763259 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":93,"threads_started":1}
I20260928 06:40:21.764688 23908 raft_consensus.cc:348] T 00000000000000000000000000000000 P bffec50127a84b4eb1d49d310e94998a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffec50127a84b4eb1d49d310e94998a" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35053 } }
I20260928 06:40:21.764823 23908 raft_consensus.cc:374] T 00000000000000000000000000000000 P bffec50127a84b4eb1d49d310e94998a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:21.764847 23908 raft_consensus.cc:729] T 00000000000000000000000000000000 P bffec50127a84b4eb1d49d310e94998a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bffec50127a84b4eb1d49d310e94998a, State: Initialized, Role: FOLLOWER
I20260928 06:40:21.764967 23908 consensus_queue.cc:260] T 00000000000000000000000000000000 P bffec50127a84b4eb1d49d310e94998a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffec50127a84b4eb1d49d310e94998a" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35053 } }
I20260928 06:40:21.765300 23908 raft_consensus.cc:388] T 00000000000000000000000000000000 P bffec50127a84b4eb1d49d310e94998a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:40:21.765367 23908 raft_consensus.cc:482] T 00000000000000000000000000000000 P bffec50127a84b4eb1d49d310e94998a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:40:21.765506 23908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bffec50127a84b4eb1d49d310e94998a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:21.766041 23908 raft_consensus.cc:504] T 00000000000000000000000000000000 P bffec50127a84b4eb1d49d310e94998a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffec50127a84b4eb1d49d310e94998a" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35053 } }
I20260928 06:40:21.766134 23908 leader_election.cc:302] T 00000000000000000000000000000000 P bffec50127a84b4eb1d49d310e94998a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bffec50127a84b4eb1d49d310e94998a; no voters:
I20260928 06:40:21.766292 23908 leader_election.cc:288] T 00000000000000000000000000000000 P bffec50127a84b4eb1d49d310e94998a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:40:21.766358 23913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bffec50127a84b4eb1d49d310e94998a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:21.766515 23913 raft_consensus.cc:686] T 00000000000000000000000000000000 P bffec50127a84b4eb1d49d310e94998a [term 1 LEADER]: Becoming Leader. State: Replica: bffec50127a84b4eb1d49d310e94998a, State: Running, Role: LEADER
I20260928 06:40:21.766556 23908 sys_catalog.cc:564] T 00000000000000000000000000000000 P bffec50127a84b4eb1d49d310e94998a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:40:21.766609 23913 consensus_queue.cc:237] T 00000000000000000000000000000000 P bffec50127a84b4eb1d49d310e94998a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffec50127a84b4eb1d49d310e94998a" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35053 } }
I20260928 06:40:21.766953 23914 sys_catalog.cc:455] T 00000000000000000000000000000000 P bffec50127a84b4eb1d49d310e94998a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bffec50127a84b4eb1d49d310e94998a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffec50127a84b4eb1d49d310e94998a" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35053 } } }
I20260928 06:40:21.767061 23914 sys_catalog.cc:458] T 00000000000000000000000000000000 P bffec50127a84b4eb1d49d310e94998a [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:21.766954 23915 sys_catalog.cc:455] T 00000000000000000000000000000000 P bffec50127a84b4eb1d49d310e94998a [sys.catalog]: SysCatalogTable state changed. Reason: New leader bffec50127a84b4eb1d49d310e94998a. Latest consensus state: current_term: 1 leader_uuid: "bffec50127a84b4eb1d49d310e94998a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bffec50127a84b4eb1d49d310e94998a" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35053 } } }
I20260928 06:40:21.767282 23915 sys_catalog.cc:458] T 00000000000000000000000000000000 P bffec50127a84b4eb1d49d310e94998a [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:21.767309 23921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:40:21.768016 23921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:40:21.769524 23921 catalog_manager.cc:1347] Generated new cluster ID: 764cc1827c144dae8828861a36da110d
I20260928 06:40:21.769577 23921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:40:21.781229 23921 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:40:21.781386 23921 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:40:21.786957 23921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bffec50127a84b4eb1d49d310e94998a: Generated new TSK 0
I20260928 06:40:21.787133 23921 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:40:21.789662 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.65:0
--local_ip_for_outbound_sockets=127.18.211.65
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35053
--builtin_ntp_servers=127.18.211.84:35905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 06:40:21.883596 23932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:21.883800 23932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:21.883828 23932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:21.884974 23932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:21.885011 23932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:21.885044 23932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.65
I20260928 06:40:21.886317 23932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:35905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.65
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:21.886528 23932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:21.886786 23932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:21.892380 23937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:21.893020 23940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:21.893083 23938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:21.900384 23932 server_base.cc:1034] running on GCE node
I20260928 06:40:21.900651 23932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:21.900956 23932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:21.902104 23932 hybrid_clock.cc:648] HybridClock initialized: now 1790577621902081 us; error 38 us; skew 500 ppm
I20260928 06:40:21.903677 23932 webserver.cc:460] Webserver started at http://127.18.211.65:38307/ using document root <none> and password file <none>
I20260928 06:40:21.903927 23932 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:21.903981 23932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:21.904100 23932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:21.905519 23932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-0/data/instance:
uuid: "d60f358ef3ce4e8787e378ae72f72c29"
format_stamp: "Formatted at 2026-09-28 06:40:21 on dist-test-slave-b28r"
server_key: "48dac406a78c37f3ad7e7fcf96f298dc"
server_key_iv: "deefa16e6a9279c6e832753eb1e4772c"
server_key_version: "encryptionkey@0"
I20260928 06:40:21.905915 23932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance:
uuid: "d60f358ef3ce4e8787e378ae72f72c29"
format_stamp: "Formatted at 2026-09-28 06:40:21 on dist-test-slave-b28r"
server_key: "48dac406a78c37f3ad7e7fcf96f298dc"
server_key_iv: "deefa16e6a9279c6e832753eb1e4772c"
server_key_version: "encryptionkey@0"
I20260928 06:40:21.907559 23932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260928 06:40:21.908340 23946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:21.908478 23932 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:40:21.908560 23932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-0/wal
uuid: "d60f358ef3ce4e8787e378ae72f72c29"
format_stamp: "Formatted at 2026-09-28 06:40:21 on dist-test-slave-b28r"
server_key: "48dac406a78c37f3ad7e7fcf96f298dc"
server_key_iv: "deefa16e6a9279c6e832753eb1e4772c"
server_key_version: "encryptionkey@0"
I20260928 06:40:21.908627 23932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:21.911865 23932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:21.912114 23932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:21.912235 23932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:21.912431 23932 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:21.912719 23932 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:21.912750 23932 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:21.912767 23932 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:21.912791 23932 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:21.918098 23932 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.65:39305
I20260928 06:40:21.918150 24059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.65:39305 every 8 connection(s)
I20260928 06:40:21.918428 23932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
I20260928 06:40:21.921175 23911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:21.919143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:45735 (local address 127.18.211.126:35053)
0928 06:40:21.919195 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:21.919197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:21.919456 (+ 259us) server_negotiation.cc:407] Connection header received
0928 06:40:21.919903 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:21.919904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:21.919910 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:21.919969 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:21.920996 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:21.920997 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:21.920998 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:40:21.921002 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:21.921015 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:40:21.921019 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:21.921059 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:21.921110 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 06:40:21.921122 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:40:21.921329 24062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:21.919169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35053 (local address 127.18.211.65:45735)
0928 06:40:21.919339 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:21.919371 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:21.919801 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:21.920025 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:21.920028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:21.920037 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:21.920866 (+ 829us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:40:21.920867 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:21.920893 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:40:21.920920 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:21.921059 (+ 139us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:21.921061 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:21.921063 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:40:21.921110 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 06:40:21.921154 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
I20260928 06:40:21.921774 24060 heartbeater.cc:347] Connected to a master server at 127.18.211.126:35053
I20260928 06:40:21.921851 24060 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:21.921958 24060 heartbeater.cc:511] Master 127.18.211.126:35053 requested a full tablet report, sending...
I20260928 06:40:21.922441 23872 ts_manager.cc:194] Registered new tserver with Master: d60f358ef3ce4e8787e378ae72f72c29 (127.18.211.65:39305)
I20260928 06:40:21.924898 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 23932
I20260928 06:40:21.924988 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance
I20260928 06:40:21.925145 19277 external_mini_cluster.cc:1428] Setting key 62f0ee2c8da61dd9875455e5bcd8b2f6
I20260928 06:40:21.926169 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.66:0
--local_ip_for_outbound_sockets=127.18.211.66
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35053
--builtin_ntp_servers=127.18.211.84:35905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 06:40:22.006457 24063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:22.006668 24063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:22.006693 24063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:22.007897 24063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:22.007942 24063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:22.007972 24063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.66
I20260928 06:40:22.009353 24063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:35905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.66
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:22.009582 24063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:22.009883 24063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:22.014816 24069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:22.014806 24068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:22.014890 24071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:22.015023 24063 server_base.cc:1034] running on GCE node
I20260928 06:40:22.015234 24063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:22.015453 24063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:22.016597 24063 hybrid_clock.cc:648] HybridClock initialized: now 1790577622016576 us; error 35 us; skew 500 ppm
I20260928 06:40:22.017859 24063 webserver.cc:460] Webserver started at http://127.18.211.66:34635/ using document root <none> and password file <none>
I20260928 06:40:22.018085 24063 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:22.018129 24063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:22.018321 24063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:22.019457 24063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-1/data/instance:
uuid: "615f0bfabd5a4bdb80e50f8f9262ef33"
format_stamp: "Formatted at 2026-09-28 06:40:22 on dist-test-slave-b28r"
server_key: "5b40be2cfb55d2f7afba4729361cb267"
server_key_iv: "263a4a1adf1fa8b33f78cf80a6750bb1"
server_key_version: "encryptionkey@0"
I20260928 06:40:22.019826 24063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance:
uuid: "615f0bfabd5a4bdb80e50f8f9262ef33"
format_stamp: "Formatted at 2026-09-28 06:40:22 on dist-test-slave-b28r"
server_key: "5b40be2cfb55d2f7afba4729361cb267"
server_key_iv: "263a4a1adf1fa8b33f78cf80a6750bb1"
server_key_version: "encryptionkey@0"
I20260928 06:40:22.021160 24063 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:22.021894 24077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:22.022059 24063 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:40:22.022137 24063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-1/wal
uuid: "615f0bfabd5a4bdb80e50f8f9262ef33"
format_stamp: "Formatted at 2026-09-28 06:40:22 on dist-test-slave-b28r"
server_key: "5b40be2cfb55d2f7afba4729361cb267"
server_key_iv: "263a4a1adf1fa8b33f78cf80a6750bb1"
server_key_version: "encryptionkey@0"
I20260928 06:40:22.022200 24063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:22.025504 24063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:22.025738 24063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:22.025851 24063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:22.026048 24063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:22.026340 24063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:22.026371 24063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:22.026396 24063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:22.026422 24063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:22.031764 24063 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.66:33489
I20260928 06:40:22.031810 24190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.66:33489 every 8 connection(s)
I20260928 06:40:22.032092 24063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
I20260928 06:40:22.034951 23911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:22.032811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:39035 (local address 127.18.211.126:35053)
0928 06:40:22.032875 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:22.032878 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:22.033074 (+ 196us) server_negotiation.cc:407] Connection header received
0928 06:40:22.033581 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:22.033583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:22.033588 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:22.033649 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:22.034749 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:22.034750 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:22.034750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:40:22.034754 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:22.034769 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:40:22.034773 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:22.034826 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:22.034887 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 06:40:22.034898 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 06:40:22.035140 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:22.032800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35053 (local address 127.18.211.66:39035)
0928 06:40:22.032968 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:22.033000 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:22.033476 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:22.033698 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:22.033701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:22.033708 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:22.034583 (+ 875us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:40:22.034584 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:22.034611 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:40:22.034638 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:22.034825 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:22.034828 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:22.034830 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:40:22.034885 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 06:40:22.034929 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260928 06:40:22.035626 24191 heartbeater.cc:347] Connected to a master server at 127.18.211.126:35053
I20260928 06:40:22.035696 24191 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:22.035844 24191 heartbeater.cc:511] Master 127.18.211.126:35053 requested a full tablet report, sending...
I20260928 06:40:22.036193 23872 ts_manager.cc:194] Registered new tserver with Master: 615f0bfabd5a4bdb80e50f8f9262ef33 (127.18.211.66:33489)
I20260928 06:40:22.040587 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 24063
I20260928 06:40:22.040670 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance
I20260928 06:40:22.040838 19277 external_mini_cluster.cc:1428] Setting key 716a9406d17ff8dd85906d031c36984d
I20260928 06:40:22.041730 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.67:0
--local_ip_for_outbound_sockets=127.18.211.67
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35053
--builtin_ntp_servers=127.18.211.84:35905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 06:40:22.120308 24194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:22.120517 24194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:22.120539 24194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:22.121640 24194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:22.121673 24194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:22.121696 24194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.67
I20260928 06:40:22.122954 24194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:35905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24194
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.67
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:22.123178 24194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:22.123437 24194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:22.128257 24202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:22.128270 24200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:22.128367 24199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:22.128386 24194 server_base.cc:1034] running on GCE node
I20260928 06:40:22.128711 24194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:22.128933 24194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:22.130082 24194 hybrid_clock.cc:648] HybridClock initialized: now 1790577622130067 us; error 33 us; skew 500 ppm
I20260928 06:40:22.131414 24194 webserver.cc:460] Webserver started at http://127.18.211.67:43151/ using document root <none> and password file <none>
I20260928 06:40:22.131661 24194 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:22.131713 24194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:22.131844 24194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:22.132893 24194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-2/data/instance:
uuid: "60e6a00bad844cd6ac79d5953af03b4b"
format_stamp: "Formatted at 2026-09-28 06:40:22 on dist-test-slave-b28r"
server_key: "7635c117031924350fcf3fbbcf6903ec"
server_key_iv: "23afaa37e1dcb8ee87089232f4d695cd"
server_key_version: "encryptionkey@0"
I20260928 06:40:22.133224 24194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance:
uuid: "60e6a00bad844cd6ac79d5953af03b4b"
format_stamp: "Formatted at 2026-09-28 06:40:22 on dist-test-slave-b28r"
server_key: "7635c117031924350fcf3fbbcf6903ec"
server_key_iv: "23afaa37e1dcb8ee87089232f4d695cd"
server_key_version: "encryptionkey@0"
I20260928 06:40:22.134456 24194 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:40:22.135180 24208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:22.135334 24194 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:40:22.135421 24194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-2/wal
uuid: "60e6a00bad844cd6ac79d5953af03b4b"
format_stamp: "Formatted at 2026-09-28 06:40:22 on dist-test-slave-b28r"
server_key: "7635c117031924350fcf3fbbcf6903ec"
server_key_iv: "23afaa37e1dcb8ee87089232f4d695cd"
server_key_version: "encryptionkey@0"
I20260928 06:40:22.135488 24194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:22.138640 24194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:22.138890 24194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:22.139004 24194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:22.139211 24194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:22.139529 24194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:22.139564 24194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:22.139590 24194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:22.139616 24194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:22.145021 24194 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.67:34999
I20260928 06:40:22.145073 24321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.67:34999 every 8 connection(s)
I20260928 06:40:22.145352 24194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
I20260928 06:40:22.145924 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 24194
I20260928 06:40:22.146045 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance
I20260928 06:40:22.146241 19277 external_mini_cluster.cc:1428] Setting key 5c1feb3d29330e1f25e51591e54329c6
I20260928 06:40:22.148356 23911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:22.146052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:38695 (local address 127.18.211.126:35053)
0928 06:40:22.146104 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:22.146106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:22.146362 (+ 256us) server_negotiation.cc:407] Connection header received
0928 06:40:22.146904 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:22.146906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:22.146913 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:22.146965 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:22.148125 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:22.148126 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:22.148127 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:40:22.148131 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:22.148147 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:40:22.148152 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:22.148201 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:22.148272 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 06:40:22.148286 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:40:22.148553 24324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:22.146052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35053 (local address 127.18.211.67:38695)
0928 06:40:22.146243 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:22.146275 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:22.146805 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:22.147035 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:22.147039 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:22.147046 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:22.147979 (+ 933us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:40:22.147979 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:22.148007 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:40:22.148035 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:22.148206 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:22.148211 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:22.148215 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:40:22.148284 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 06:40:22.148345 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":98,"threads_started":1}
I20260928 06:40:22.149077 24322 heartbeater.cc:347] Connected to a master server at 127.18.211.126:35053
I20260928 06:40:22.149155 24322 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:22.149267 24322 heartbeater.cc:511] Master 127.18.211.126:35053 requested a full tablet report, sending...
I20260928 06:40:22.149680 23872 ts_manager.cc:194] Registered new tserver with Master: 60e6a00bad844cd6ac79d5953af03b4b (127.18.211.67:34999)
I20260928 06:40:22.157855 19277 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:40:22.160209 23911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:22.159181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33816 (local address 127.18.211.126:35053)
0928 06:40:22.159248 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:22.159250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:22.159412 (+ 162us) server_negotiation.cc:407] Connection header received
0928 06:40:22.159456 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:22.159457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:22.159461 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:22.159508 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:22.160022 (+ 514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:22.160024 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:22.160024 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:40:22.160028 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:22.160043 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:40:22.160047 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:22.160090 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:22.160129 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 06:40:22.160140 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 06:40:22.161614 23872 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:40:22.163208 23872 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:40:22.166740 24334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:22.165204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:34999 (local address 127.0.0.1:49406)
0928 06:40:22.165518 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:22.165569 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:22.165629 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:22.165804 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:22.165809 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:22.165816 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:22.166297 (+ 481us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:40:22.166297 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:22.166315 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:40:22.166339 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:22.166577 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:22.166583 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:22.166587 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:40:22.166648 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 06:40:22.166662 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":69,"threads_started":1}
I20260928 06:40:22.166786 24332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:22.165293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49406 (local address 127.18.211.67:34999)
0928 06:40:22.165470 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:22.165475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:22.165643 (+ 168us) server_negotiation.cc:407] Connection header received
0928 06:40:22.165681 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:22.165684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:22.165694 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:22.165796 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:22.166399 (+ 603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:22.166404 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:22.166404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:40:22.166407 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:22.166443 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:40:22.166462 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:22.166580 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:22.166680 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 06:40:22.166695 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":96,"threads_started":1}
I20260928 06:40:22.167294 24331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:22.165405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:33489 (local address 127.0.0.1:34008)
0928 06:40:22.165680 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:22.165688 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:22.165748 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:22.166367 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:22.166368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:22.166371 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:22.166828 (+ 457us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:40:22.166828 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:22.166835 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:40:22.166847 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:22.167187 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:22.167187 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:22.167188 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 06:40:22.167223 (+ 35us) client_negotiation.cc:240] Negotiation successful
0928 06:40:22.167235 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":57,"threads_started":1}
I20260928 06:40:22.167335 24335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:22.165592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34008 (local address 127.18.211.66:33489)
0928 06:40:22.165764 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:22.165768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:22.165789 (+ 21us) server_negotiation.cc:407] Connection header received
0928 06:40:22.165812 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:22.165814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:22.165822 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:22.165881 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:22.166933 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:22.166940 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:22.166941 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:40:22.166944 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:22.166975 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:40:22.166997 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:22.167041 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:22.167247 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 06:40:22.167261 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":98,"threads_started":1}
I20260928 06:40:22.167711 24256 tablet_service.cc:1467] Processing CreateTablet for tablet 776ccc69af0249759fccfae0fc7d13ee (DEFAULT_TABLE table=test-table [id=d983ad850079437f889b1c15d2e0c276]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:22.167950 24333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:22.165319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53938 (local address 127.18.211.65:39305)
0928 06:40:22.165492 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:22.165495 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:22.165632 (+ 137us) server_negotiation.cc:407] Connection header received
0928 06:40:22.165862 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:22.165871 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:22.165881 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:22.165929 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:22.167174 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:22.167183 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:22.167183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:40:22.167187 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:22.167228 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:40:22.167250 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:22.167309 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:22.167842 (+ 533us) server_negotiation.cc:299] Negotiation successful
0928 06:40:22.167859 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":78,"threads_started":1}
I20260928 06:40:22.167953 24330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:22.165207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:39305 (local address 127.0.0.1:53938)
0928 06:40:22.165539 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:22.165570 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:22.165790 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:22.166362 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:22.166362 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:22.166370 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:22.167099 (+ 729us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:40:22.167099 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:22.167105 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:40:22.167115 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:22.167754 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:22.167754 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:22.167755 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 06:40:22.167817 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 06:40:22.167829 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":87,"threads_started":1}
I20260928 06:40:22.168107 24256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 776ccc69af0249759fccfae0fc7d13ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:22.168020 24125 tablet_service.cc:1467] Processing CreateTablet for tablet 776ccc69af0249759fccfae0fc7d13ee (DEFAULT_TABLE table=test-table [id=d983ad850079437f889b1c15d2e0c276]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:22.168336 24125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 776ccc69af0249759fccfae0fc7d13ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:22.170703 24336 tablet_bootstrap.cc:492] T 776ccc69af0249759fccfae0fc7d13ee P 615f0bfabd5a4bdb80e50f8f9262ef33: Bootstrap starting.
I20260928 06:40:22.171226 24337 tablet_bootstrap.cc:492] T 776ccc69af0249759fccfae0fc7d13ee P 60e6a00bad844cd6ac79d5953af03b4b: Bootstrap starting.
I20260928 06:40:22.171417 24336 tablet_bootstrap.cc:654] T 776ccc69af0249759fccfae0fc7d13ee P 615f0bfabd5a4bdb80e50f8f9262ef33: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:22.171317 23994 tablet_service.cc:1467] Processing CreateTablet for tablet 776ccc69af0249759fccfae0fc7d13ee (DEFAULT_TABLE table=test-table [id=d983ad850079437f889b1c15d2e0c276]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:22.171624 23994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 776ccc69af0249759fccfae0fc7d13ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:22.171736 24336 log.cc:824] T 776ccc69af0249759fccfae0fc7d13ee P 615f0bfabd5a4bdb80e50f8f9262ef33: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:22.171849 24337 tablet_bootstrap.cc:654] T 776ccc69af0249759fccfae0fc7d13ee P 60e6a00bad844cd6ac79d5953af03b4b: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:22.172120 24337 log.cc:824] T 776ccc69af0249759fccfae0fc7d13ee P 60e6a00bad844cd6ac79d5953af03b4b: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:22.172472 24336 tablet_bootstrap.cc:492] T 776ccc69af0249759fccfae0fc7d13ee P 615f0bfabd5a4bdb80e50f8f9262ef33: No bootstrap required, opened a new log
I20260928 06:40:22.172530 24336 ts_tablet_manager.cc:1397] T 776ccc69af0249759fccfae0fc7d13ee P 615f0bfabd5a4bdb80e50f8f9262ef33: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:40:22.172811 24337 tablet_bootstrap.cc:492] T 776ccc69af0249759fccfae0fc7d13ee P 60e6a00bad844cd6ac79d5953af03b4b: No bootstrap required, opened a new log
I20260928 06:40:22.172881 24337 ts_tablet_manager.cc:1397] T 776ccc69af0249759fccfae0fc7d13ee P 60e6a00bad844cd6ac79d5953af03b4b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:40:22.174378 24336 raft_consensus.cc:348] T 776ccc69af0249759fccfae0fc7d13ee P 615f0bfabd5a4bdb80e50f8f9262ef33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e6a00bad844cd6ac79d5953af03b4b" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34999 } } peers { permanent_uuid: "d60f358ef3ce4e8787e378ae72f72c29" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 39305 } } peers { permanent_uuid: "615f0bfabd5a4bdb80e50f8f9262ef33" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33489 } }
I20260928 06:40:22.174508 24336 raft_consensus.cc:374] T 776ccc69af0249759fccfae0fc7d13ee P 615f0bfabd5a4bdb80e50f8f9262ef33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:22.174540 24336 raft_consensus.cc:729] T 776ccc69af0249759fccfae0fc7d13ee P 615f0bfabd5a4bdb80e50f8f9262ef33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615f0bfabd5a4bdb80e50f8f9262ef33, State: Initialized, Role: FOLLOWER
I20260928 06:40:22.174685 24336 consensus_queue.cc:260] T 776ccc69af0249759fccfae0fc7d13ee P 615f0bfabd5a4bdb80e50f8f9262ef33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e6a00bad844cd6ac79d5953af03b4b" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34999 } } peers { permanent_uuid: "d60f358ef3ce4e8787e378ae72f72c29" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 39305 } } peers { permanent_uuid: "615f0bfabd5a4bdb80e50f8f9262ef33" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33489 } }
I20260928 06:40:22.174947 24336 ts_tablet_manager.cc:1428] T 776ccc69af0249759fccfae0fc7d13ee P 615f0bfabd5a4bdb80e50f8f9262ef33: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:40:22.174926 24340 tablet_bootstrap.cc:492] T 776ccc69af0249759fccfae0fc7d13ee P d60f358ef3ce4e8787e378ae72f72c29: Bootstrap starting.
I20260928 06:40:22.175063 24191 heartbeater.cc:503] Master 127.18.211.126:35053 was elected leader, sending a full tablet report...
I20260928 06:40:22.175670 24340 tablet_bootstrap.cc:654] T 776ccc69af0249759fccfae0fc7d13ee P d60f358ef3ce4e8787e378ae72f72c29: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:22.175618 24337 raft_consensus.cc:348] T 776ccc69af0249759fccfae0fc7d13ee P 60e6a00bad844cd6ac79d5953af03b4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e6a00bad844cd6ac79d5953af03b4b" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34999 } } peers { permanent_uuid: "d60f358ef3ce4e8787e378ae72f72c29" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 39305 } } peers { permanent_uuid: "615f0bfabd5a4bdb80e50f8f9262ef33" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33489 } }
I20260928 06:40:22.175761 24337 raft_consensus.cc:374] T 776ccc69af0249759fccfae0fc7d13ee P 60e6a00bad844cd6ac79d5953af03b4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:22.175804 24337 raft_consensus.cc:729] T 776ccc69af0249759fccfae0fc7d13ee P 60e6a00bad844cd6ac79d5953af03b4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60e6a00bad844cd6ac79d5953af03b4b, State: Initialized, Role: FOLLOWER
I20260928 06:40:22.176009 24340 log.cc:824] T 776ccc69af0249759fccfae0fc7d13ee P d60f358ef3ce4e8787e378ae72f72c29: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:22.175994 24337 consensus_queue.cc:260] T 776ccc69af0249759fccfae0fc7d13ee P 60e6a00bad844cd6ac79d5953af03b4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e6a00bad844cd6ac79d5953af03b4b" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34999 } } peers { permanent_uuid: "d60f358ef3ce4e8787e378ae72f72c29" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 39305 } } peers { permanent_uuid: "615f0bfabd5a4bdb80e50f8f9262ef33" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33489 } }
I20260928 06:40:22.176290 24337 ts_tablet_manager.cc:1428] T 776ccc69af0249759fccfae0fc7d13ee P 60e6a00bad844cd6ac79d5953af03b4b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:40:22.176316 24322 heartbeater.cc:503] Master 127.18.211.126:35053 was elected leader, sending a full tablet report...
I20260928 06:40:22.177513 24340 tablet_bootstrap.cc:492] T 776ccc69af0249759fccfae0fc7d13ee P d60f358ef3ce4e8787e378ae72f72c29: No bootstrap required, opened a new log
I20260928 06:40:22.177568 24340 ts_tablet_manager.cc:1397] T 776ccc69af0249759fccfae0fc7d13ee P d60f358ef3ce4e8787e378ae72f72c29: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 06:40:22.179311 24340 raft_consensus.cc:348] T 776ccc69af0249759fccfae0fc7d13ee P d60f358ef3ce4e8787e378ae72f72c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e6a00bad844cd6ac79d5953af03b4b" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34999 } } peers { permanent_uuid: "d60f358ef3ce4e8787e378ae72f72c29" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 39305 } } peers { permanent_uuid: "615f0bfabd5a4bdb80e50f8f9262ef33" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33489 } }
I20260928 06:40:22.179432 24340 raft_consensus.cc:374] T 776ccc69af0249759fccfae0fc7d13ee P d60f358ef3ce4e8787e378ae72f72c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:22.179469 24340 raft_consensus.cc:729] T 776ccc69af0249759fccfae0fc7d13ee P d60f358ef3ce4e8787e378ae72f72c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d60f358ef3ce4e8787e378ae72f72c29, State: Initialized, Role: FOLLOWER
I20260928 06:40:22.179579 24340 consensus_queue.cc:260] T 776ccc69af0249759fccfae0fc7d13ee P d60f358ef3ce4e8787e378ae72f72c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e6a00bad844cd6ac79d5953af03b4b" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34999 } } peers { permanent_uuid: "d60f358ef3ce4e8787e378ae72f72c29" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 39305 } } peers { permanent_uuid: "615f0bfabd5a4bdb80e50f8f9262ef33" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33489 } }
I20260928 06:40:22.179788 24340 ts_tablet_manager.cc:1428] T 776ccc69af0249759fccfae0fc7d13ee P d60f358ef3ce4e8787e378ae72f72c29: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:40:22.179840 24060 heartbeater.cc:503] Master 127.18.211.126:35053 was elected leader, sending a full tablet report...
W20260928 06:40:22.282640 24192 tablet.cc:2369] T 776ccc69af0249759fccfae0fc7d13ee P 615f0bfabd5a4bdb80e50f8f9262ef33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:40:22.338459 24341 raft_consensus.cc:482] T 776ccc69af0249759fccfae0fc7d13ee P 615f0bfabd5a4bdb80e50f8f9262ef33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:40:22.338611 24341 raft_consensus.cc:504] T 776ccc69af0249759fccfae0fc7d13ee P 615f0bfabd5a4bdb80e50f8f9262ef33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e6a00bad844cd6ac79d5953af03b4b" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34999 } } peers { permanent_uuid: "d60f358ef3ce4e8787e378ae72f72c29" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 39305 } } peers { permanent_uuid: "615f0bfabd5a4bdb80e50f8f9262ef33" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33489 } }
I20260928 06:40:22.338963 24341 leader_election.cc:288] T 776ccc69af0249759fccfae0fc7d13ee P 615f0bfabd5a4bdb80e50f8f9262ef33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60e6a00bad844cd6ac79d5953af03b4b (127.18.211.67:34999), d60f358ef3ce4e8787e378ae72f72c29 (127.18.211.65:39305)
I20260928 06:40:22.340369 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:22.339190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:39305 (local address 127.18.211.66:41165)
0928 06:40:22.339255 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:22.339268 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:22.339308 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:22.339473 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:22.339475 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:22.339478 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:22.339981 (+ 503us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:40:22.339981 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:22.339991 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:40:22.340006 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:22.340209 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:22.340210 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:22.340211 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 06:40:22.340277 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 06:40:22.340292 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 06:40:22.340368 24333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:22.339267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:41165 (local address 127.18.211.65:39305)
0928 06:40:22.339312 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:22.339313 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:22.339364 (+ 51us) server_negotiation.cc:407] Connection header received
0928 06:40:22.339390 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:22.339391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:22.339396 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:22.339451 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:22.340102 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:22.340104 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:22.340104 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:40:22.340110 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:22.340130 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:40:22.340135 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:22.340201 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:22.340279 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 06:40:22.340292 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:40:22.340498 24345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:22.339190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:34999 (local address 127.18.211.66:55433)
0928 06:40:22.339412 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:22.339420 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:22.339464 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:22.339592 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:22.339593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:22.339595 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:22.340132 (+ 537us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:40:22.340133 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:22.340139 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:40:22.340150 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:22.340329 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:22.340330 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:22.340331 (+ 1us) client_negotiation.cc:769] Sending connection context
0928 06:40:22.340412 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 06:40:22.340425 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":14,"thread_start_us":126,"threads_started":1}
I20260928 06:40:22.340498 24332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:22.339257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:55433 (local address 127.18.211.67:34999)
0928 06:40:22.339306 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:22.339308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:22.339499 (+ 191us) server_negotiation.cc:407] Connection header received
0928 06:40:22.339530 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:22.339531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:22.339536 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:22.339579 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:22.340230 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:22.340232 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:22.340232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:40:22.340238 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:22.340258 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:40:22.340263 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:22.340319 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:22.340417 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 06:40:22.340430 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:40:22.340746 24014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776ccc69af0249759fccfae0fc7d13ee" candidate_uuid: "615f0bfabd5a4bdb80e50f8f9262ef33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d60f358ef3ce4e8787e378ae72f72c29" is_pre_election: true
I20260928 06:40:22.340806 24276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776ccc69af0249759fccfae0fc7d13ee" candidate_uuid: "615f0bfabd5a4bdb80e50f8f9262ef33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60e6a00bad844cd6ac79d5953af03b4b" is_pre_election: true
I20260928 06:40:22.340917 24014 raft_consensus.cc:2454] T 776ccc69af0249759fccfae0fc7d13ee P d60f358ef3ce4e8787e378ae72f72c29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 615f0bfabd5a4bdb80e50f8f9262ef33 in term 0.
I20260928 06:40:22.340937 24276 raft_consensus.cc:2454] T 776ccc69af0249759fccfae0fc7d13ee P 60e6a00bad844cd6ac79d5953af03b4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 615f0bfabd5a4bdb80e50f8f9262ef33 in term 0.
I20260928 06:40:22.341194 24081 leader_election.cc:302] T 776ccc69af0249759fccfae0fc7d13ee P 615f0bfabd5a4bdb80e50f8f9262ef33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615f0bfabd5a4bdb80e50f8f9262ef33, d60f358ef3ce4e8787e378ae72f72c29; no voters:
I20260928 06:40:22.341346 24341 raft_consensus.cc:2781] T 776ccc69af0249759fccfae0fc7d13ee P 615f0bfabd5a4bdb80e50f8f9262ef33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:40:22.341399 24341 raft_consensus.cc:482] T 776ccc69af0249759fccfae0fc7d13ee P 615f0bfabd5a4bdb80e50f8f9262ef33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:40:22.341416 24341 raft_consensus.cc:3037] T 776ccc69af0249759fccfae0fc7d13ee P 615f0bfabd5a4bdb80e50f8f9262ef33 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:22.342160 24341 raft_consensus.cc:504] T 776ccc69af0249759fccfae0fc7d13ee P 615f0bfabd5a4bdb80e50f8f9262ef33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e6a00bad844cd6ac79d5953af03b4b" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34999 } } peers { permanent_uuid: "d60f358ef3ce4e8787e378ae72f72c29" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 39305 } } peers { permanent_uuid: "615f0bfabd5a4bdb80e50f8f9262ef33" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33489 } }
I20260928 06:40:22.342298 24341 leader_election.cc:288] T 776ccc69af0249759fccfae0fc7d13ee P 615f0bfabd5a4bdb80e50f8f9262ef33 [CANDIDATE]: Term 1 election: Requested vote from peers 60e6a00bad844cd6ac79d5953af03b4b (127.18.211.67:34999), d60f358ef3ce4e8787e378ae72f72c29 (127.18.211.65:39305)
I20260928 06:40:22.342449 24014 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776ccc69af0249759fccfae0fc7d13ee" candidate_uuid: "615f0bfabd5a4bdb80e50f8f9262ef33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d60f358ef3ce4e8787e378ae72f72c29"
I20260928 06:40:22.342525 24276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "776ccc69af0249759fccfae0fc7d13ee" candidate_uuid: "615f0bfabd5a4bdb80e50f8f9262ef33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60e6a00bad844cd6ac79d5953af03b4b"
I20260928 06:40:22.342533 24014 raft_consensus.cc:3037] T 776ccc69af0249759fccfae0fc7d13ee P d60f358ef3ce4e8787e378ae72f72c29 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:22.342597 24276 raft_consensus.cc:3037] T 776ccc69af0249759fccfae0fc7d13ee P 60e6a00bad844cd6ac79d5953af03b4b [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:22.343391 24276 raft_consensus.cc:2454] T 776ccc69af0249759fccfae0fc7d13ee P 60e6a00bad844cd6ac79d5953af03b4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615f0bfabd5a4bdb80e50f8f9262ef33 in term 1.
I20260928 06:40:22.343400 24014 raft_consensus.cc:2454] T 776ccc69af0249759fccfae0fc7d13ee P d60f358ef3ce4e8787e378ae72f72c29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 615f0bfabd5a4bdb80e50f8f9262ef33 in term 1.
I20260928 06:40:22.343554 24081 leader_election.cc:302] T 776ccc69af0249759fccfae0fc7d13ee P 615f0bfabd5a4bdb80e50f8f9262ef33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 615f0bfabd5a4bdb80e50f8f9262ef33, d60f358ef3ce4e8787e378ae72f72c29; no voters:
I20260928 06:40:22.343626 24341 raft_consensus.cc:2781] T 776ccc69af0249759fccfae0fc7d13ee P 615f0bfabd5a4bdb80e50f8f9262ef33 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:22.343765 24341 raft_consensus.cc:686] T 776ccc69af0249759fccfae0fc7d13ee P 615f0bfabd5a4bdb80e50f8f9262ef33 [term 1 LEADER]: Becoming Leader. State: Replica: 615f0bfabd5a4bdb80e50f8f9262ef33, State: Running, Role: LEADER
I20260928 06:40:22.343838 24341 consensus_queue.cc:237] T 776ccc69af0249759fccfae0fc7d13ee P 615f0bfabd5a4bdb80e50f8f9262ef33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e6a00bad844cd6ac79d5953af03b4b" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34999 } } peers { permanent_uuid: "d60f358ef3ce4e8787e378ae72f72c29" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 39305 } } peers { permanent_uuid: "615f0bfabd5a4bdb80e50f8f9262ef33" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33489 } }
I20260928 06:40:22.344681 23872 catalog_manager.cc:5495] T 776ccc69af0249759fccfae0fc7d13ee P 615f0bfabd5a4bdb80e50f8f9262ef33 reported cstate change: term changed from 0 to 1, leader changed from <none> to 615f0bfabd5a4bdb80e50f8f9262ef33 (127.18.211.66). New cstate: current_term: 1 leader_uuid: "615f0bfabd5a4bdb80e50f8f9262ef33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60e6a00bad844cd6ac79d5953af03b4b" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d60f358ef3ce4e8787e378ae72f72c29" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 39305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "615f0bfabd5a4bdb80e50f8f9262ef33" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33489 } health_report { overall_health: HEALTHY } } }
I20260928 06:40:22.350375 24335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:22.349570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34012 (local address 127.18.211.66:33489)
0928 06:40:22.349619 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:22.349621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:22.349649 (+ 28us) server_negotiation.cc:407] Connection header received
0928 06:40:22.349708 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:22.349708 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:22.349719 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:22.349784 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:22.350208 (+ 424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:22.350209 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:22.350209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:40:22.350212 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:22.350225 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:40:22.350228 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:22.350265 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:22.350303 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 06:40:22.350313 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:40:22.352007 24276 raft_consensus.cc:1261] T 776ccc69af0249759fccfae0fc7d13ee P 60e6a00bad844cd6ac79d5953af03b4b [term 1 FOLLOWER]: Refusing update from remote peer 615f0bfabd5a4bdb80e50f8f9262ef33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:40:22.352027 24014 raft_consensus.cc:1261] T 776ccc69af0249759fccfae0fc7d13ee P d60f358ef3ce4e8787e378ae72f72c29 [term 1 FOLLOWER]: Refusing update from remote peer 615f0bfabd5a4bdb80e50f8f9262ef33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:40:22.352319 24341 consensus_queue.cc:1035] T 776ccc69af0249759fccfae0fc7d13ee P 615f0bfabd5a4bdb80e50f8f9262ef33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d60f358ef3ce4e8787e378ae72f72c29" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 39305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:40:22.352469 24346 consensus_queue.cc:1035] T 776ccc69af0249759fccfae0fc7d13ee P 615f0bfabd5a4bdb80e50f8f9262ef33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60e6a00bad844cd6ac79d5953af03b4b" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:40:22.354656 24350 mvcc.cc:205] Tried to move back new op lower bound from 7334205941151432704 to 7334205941120942080. Current Snapshot: MvccSnapshot[applied={T|T < 7334205941151432704}]
I20260928 06:40:22.355247 24351 mvcc.cc:205] Tried to move back new op lower bound from 7334205941151432704 to 7334205941120942080. Current Snapshot: MvccSnapshot[applied={T|T < 7334205941151432704}]
I20260928 06:40:22.357241 23872 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:40:22.357334 23872 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:40:22.358028 23872 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bffec50127a84b4eb1d49d310e94998a: Sending DeleteTablet for 3 replicas of tablet 776ccc69af0249759fccfae0fc7d13ee
I20260928 06:40:22.358317 24256 tablet_service.cc:1514] Processing DeleteTablet for tablet 776ccc69af0249759fccfae0fc7d13ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:22 UTC) from {username='slave'} at 127.0.0.1:49406
I20260928 06:40:22.358373 23994 tablet_service.cc:1514] Processing DeleteTablet for tablet 776ccc69af0249759fccfae0fc7d13ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:22 UTC) from {username='slave'} at 127.0.0.1:53938
I20260928 06:40:22.358408 24125 tablet_service.cc:1514] Processing DeleteTablet for tablet 776ccc69af0249759fccfae0fc7d13ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:22 UTC) from {username='slave'} at 127.0.0.1:34008
I20260928 06:40:22.358525 24358 tablet_replica.cc:330] T 776ccc69af0249759fccfae0fc7d13ee P 60e6a00bad844cd6ac79d5953af03b4b: stopping tablet replica
I20260928 06:40:22.358587 24360 tablet_replica.cc:330] T 776ccc69af0249759fccfae0fc7d13ee P d60f358ef3ce4e8787e378ae72f72c29: stopping tablet replica
I20260928 06:40:22.358613 24358 raft_consensus.cc:2229] T 776ccc69af0249759fccfae0fc7d13ee P 60e6a00bad844cd6ac79d5953af03b4b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:40:22.358673 24360 raft_consensus.cc:2229] T 776ccc69af0249759fccfae0fc7d13ee P d60f358ef3ce4e8787e378ae72f72c29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:40:22.358690 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 23932
I20260928 06:40:22.358709 24358 raft_consensus.cc:2258] T 776ccc69af0249759fccfae0fc7d13ee P 60e6a00bad844cd6ac79d5953af03b4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:22.359485 24358 ts_tablet_manager.cc:1905] T 776ccc69af0249759fccfae0fc7d13ee P 60e6a00bad844cd6ac79d5953af03b4b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:22.359596 24359 tablet_replica.cc:330] T 776ccc69af0249759fccfae0fc7d13ee P 615f0bfabd5a4bdb80e50f8f9262ef33: stopping tablet replica
I20260928 06:40:22.359689 24359 raft_consensus.cc:2229] T 776ccc69af0249759fccfae0fc7d13ee P 615f0bfabd5a4bdb80e50f8f9262ef33 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:40:22.359800 24359 raft_consensus.cc:2258] T 776ccc69af0249759fccfae0fc7d13ee P 615f0bfabd5a4bdb80e50f8f9262ef33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:22.360159 24359 ts_tablet_manager.cc:1905] T 776ccc69af0249759fccfae0fc7d13ee P 615f0bfabd5a4bdb80e50f8f9262ef33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:22.361016 24358 ts_tablet_manager.cc:1918] T 776ccc69af0249759fccfae0fc7d13ee P 60e6a00bad844cd6ac79d5953af03b4b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:40:22.361089 24358 log.cc:1196] T 776ccc69af0249759fccfae0fc7d13ee P 60e6a00bad844cd6ac79d5953af03b4b: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-2/wal/wals/776ccc69af0249759fccfae0fc7d13ee
I20260928 06:40:22.361320 24358 ts_tablet_manager.cc:1939] T 776ccc69af0249759fccfae0fc7d13ee P 60e6a00bad844cd6ac79d5953af03b4b: Deleting consensus metadata
I20260928 06:40:22.361749 23858 catalog_manager.cc:4841] TS 60e6a00bad844cd6ac79d5953af03b4b (127.18.211.67:34999): tablet 776ccc69af0249759fccfae0fc7d13ee (table test-table [id=d983ad850079437f889b1c15d2e0c276]) successfully deleted
I20260928 06:40:22.364318 24359 ts_tablet_manager.cc:1918] T 776ccc69af0249759fccfae0fc7d13ee P 615f0bfabd5a4bdb80e50f8f9262ef33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:40:22.364383 24359 log.cc:1196] T 776ccc69af0249759fccfae0fc7d13ee P 615f0bfabd5a4bdb80e50f8f9262ef33: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790577602522428-19277-0/minicluster-data/ts-1/wal/wals/776ccc69af0249759fccfae0fc7d13ee
I20260928 06:40:22.364634 24359 ts_tablet_manager.cc:1939] T 776ccc69af0249759fccfae0fc7d13ee P 615f0bfabd5a4bdb80e50f8f9262ef33: Deleting consensus metadata
I20260928 06:40:22.364959 23859 catalog_manager.cc:4841] TS 615f0bfabd5a4bdb80e50f8f9262ef33 (127.18.211.66:33489): tablet 776ccc69af0249759fccfae0fc7d13ee (table test-table [id=d983ad850079437f889b1c15d2e0c276]) successfully deleted
W20260928 06:40:22.365645 23859 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.211.65:39305 (error 108)
I20260928 06:40:22.365842 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 24063
I20260928 06:40:22.366560 24330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:22.365965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:39305 (local address 127.0.0.1:53946)
0928 06:40:22.366486 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:22.366530 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.65:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501}
W20260928 06:40:22.366681 23859 catalog_manager.cc:4584] TS d60f358ef3ce4e8787e378ae72f72c29 (127.18.211.65:39305): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 776ccc69af0249759fccfae0fc7d13ee: Network error: Client connection negotiation failed: client connection to 127.18.211.65:39305: connect: Connection refused (error 111)
I20260928 06:40:22.370491 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 24194
I20260928 06:40:22.374804 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 23841
2026-09-28T06:40:22Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (743 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 06:40:22.392508 19277 mini_oidc.cc:152] Starting JWKS server
I20260928 06:40:22.393520 19277 webserver.cc:460] Webserver started at https://127.0.0.1:41935/ using document root <none> and password file <none>
I20260928 06:40:22.393610 19277 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 06:40:22.393770 19277 webserver.cc:460] Webserver started at http://127.0.0.1:38431/ using document root <none> and password file <none>
2026-09-28T06:40:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:40:22Z Disabled control of system clock
I20260928 06:40:22.400967 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:40683
--webserver_interface=127.18.211.126
--webserver_port=0
--builtin_ntp_servers=127.18.211.84:41471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:40683
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41935/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577602522428-19277-0/testchainca.cert with env {}
W20260928 06:40:22.479332 24367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:22.479528 24367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:22.479548 24367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:22.480674 24367 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:40:22.480712 24367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:22.480727 24367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:22.480738 24367 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:40:22.480749 24367 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 06:40:22.480759 24367 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 06:40:22.480769 24367 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41935/jwks.json
I20260928 06:40:22.482255 24367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:41471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:40683
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:40683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41935/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577602522428-19277-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:22.482450 24367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:22.482604 24367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:22.487170 24375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:22.487178 24373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:22.487341 24367 server_base.cc:1034] running on GCE node
W20260928 06:40:22.487180 24372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:22.487829 24367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:22.488142 24367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:22.489285 24367 hybrid_clock.cc:648] HybridClock initialized: now 1790577622489275 us; error 28 us; skew 500 ppm
I20260928 06:40:22.490430 24367 webserver.cc:460] Webserver started at http://127.18.211.126:35557/ using document root <none> and password file <none>
I20260928 06:40:22.490628 24367 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:22.490674 24367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:22.490790 24367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:22.491870 24367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "6f1182cb27114eba83135d467d634415"
format_stamp: "Formatted at 2026-09-28 06:40:22 on dist-test-slave-b28r"
server_key: "91f4695755eaaeeacd6dda5f93ee25db"
server_key_iv: "d577250154335057c384680b34f485d0"
server_key_version: "encryptionkey@0"
I20260928 06:40:22.492192 24367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "6f1182cb27114eba83135d467d634415"
format_stamp: "Formatted at 2026-09-28 06:40:22 on dist-test-slave-b28r"
server_key: "91f4695755eaaeeacd6dda5f93ee25db"
server_key_iv: "d577250154335057c384680b34f485d0"
server_key_version: "encryptionkey@0"
I20260928 06:40:22.493412 24367 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:40:22.494186 24381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:22.494385 24367 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:22.494475 24367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "6f1182cb27114eba83135d467d634415"
format_stamp: "Formatted at 2026-09-28 06:40:22 on dist-test-slave-b28r"
server_key: "91f4695755eaaeeacd6dda5f93ee25db"
server_key_iv: "d577250154335057c384680b34f485d0"
server_key_version: "encryptionkey@0"
I20260928 06:40:22.494534 24367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:22.516171 24367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:22.516487 24367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:22.516602 24367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:22.520787 24367 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:40683
I20260928 06:40:22.520853 24433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:40683 every 8 connection(s)
I20260928 06:40:22.521199 24367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:40:22.522135 24434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:22.524482 24434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f1182cb27114eba83135d467d634415: Bootstrap starting.
I20260928 06:40:22.524850 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 24367
I20260928 06:40:22.524956 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:40:22.525094 24434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f1182cb27114eba83135d467d634415: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:22.525127 19277 external_mini_cluster.cc:1428] Setting key bbde437d7fc084c0e747f075b9c40ff1
I20260928 06:40:22.525391 24434 log.cc:824] T 00000000000000000000000000000000 P 6f1182cb27114eba83135d467d634415: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:22.526089 24434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f1182cb27114eba83135d467d634415: No bootstrap required, opened a new log
I20260928 06:40:22.527899 24434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f1182cb27114eba83135d467d634415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1182cb27114eba83135d467d634415" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 40683 } }
I20260928 06:40:22.528033 24434 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f1182cb27114eba83135d467d634415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:22.528064 24434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f1182cb27114eba83135d467d634415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f1182cb27114eba83135d467d634415, State: Initialized, Role: FOLLOWER
I20260928 06:40:22.528187 24434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f1182cb27114eba83135d467d634415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1182cb27114eba83135d467d634415" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 40683 } }
I20260928 06:40:22.528364 24434 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f1182cb27114eba83135d467d634415 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:40:22.528420 24434 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f1182cb27114eba83135d467d634415 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:40:22.528476 24434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f1182cb27114eba83135d467d634415 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:22.529129 24434 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f1182cb27114eba83135d467d634415 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1182cb27114eba83135d467d634415" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 40683 } }
I20260928 06:40:22.529235 24434 leader_election.cc:302] T 00000000000000000000000000000000 P 6f1182cb27114eba83135d467d634415 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f1182cb27114eba83135d467d634415; no voters:
I20260928 06:40:22.529404 24434 leader_election.cc:288] T 00000000000000000000000000000000 P 6f1182cb27114eba83135d467d634415 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:40:22.529449 24439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f1182cb27114eba83135d467d634415 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:22.529573 24439 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f1182cb27114eba83135d467d634415 [term 1 LEADER]: Becoming Leader. State: Replica: 6f1182cb27114eba83135d467d634415, State: Running, Role: LEADER
I20260928 06:40:22.529675 24434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f1182cb27114eba83135d467d634415 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:40:22.529659 24439 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f1182cb27114eba83135d467d634415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1182cb27114eba83135d467d634415" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 40683 } }
I20260928 06:40:22.529999 24439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f1182cb27114eba83135d467d634415 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f1182cb27114eba83135d467d634415. Latest consensus state: current_term: 1 leader_uuid: "6f1182cb27114eba83135d467d634415" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1182cb27114eba83135d467d634415" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 40683 } } }
I20260928 06:40:22.530076 24439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f1182cb27114eba83135d467d634415 [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:22.530318 24440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f1182cb27114eba83135d467d634415 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f1182cb27114eba83135d467d634415" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f1182cb27114eba83135d467d634415" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 40683 } } }
I20260928 06:40:22.530464 24440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f1182cb27114eba83135d467d634415 [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:22.530614 24445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:40:22.530884 24438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:22.525610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59856 (local address 127.18.211.126:40683)
0928 06:40:22.525831 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:22.525840 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:22.525867 (+ 27us) server_negotiation.cc:407] Connection header received
0928 06:40:22.526279 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:22.526286 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:22.526347 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:22.526474 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:22.526664 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:22.527541 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:22.528490 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:22.529108 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:22.529241 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:22.529926 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:22.529930 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:22.529930 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:40:22.530135 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:22.530185 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:40:22.530202 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:22.530275 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:22.530578 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 06:40:22.530657 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":90,"threads_started":1}
I20260928 06:40:22.531354 24445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:40:22.532053 19277 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:40:22.533041 24445 catalog_manager.cc:1347] Generated new cluster ID: d4d6229b879f485cac29cbeb45f3335f
I20260928 06:40:22.533094 24445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:40:22.547806 24445 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:40:22.548395 24445 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:40:22.553259 24445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f1182cb27114eba83135d467d634415: Generated new TSK 0
I20260928 06:40:22.553435 24445 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 06:40:22.583211 24464 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 06:40:22.583379 24464 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 06:40:22.583508 24438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:22.560306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59872 (local address 127.18.211.126:40683)
0928 06:40:22.560357 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:22.560359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:22.560452 (+ 93us) server_negotiation.cc:407] Connection header received
0928 06:40:22.560559 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:22.560559 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:22.560570 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:22.560616 (+ 46us) server_negotiation.cc:226] Negotiated authn=JWT
0928 06:40:22.560791 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:22.561264 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:22.562399 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:22.562702 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:22.562781 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:22.563069 (+ 288us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 06:40:22.583203 (+ 20134us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41935/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 06:40:22.583426 (+ 223us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59872: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41935/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":29}
W20260928 06:40:22.583590 24438 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59872: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41935/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 06:40:22.583851 24463 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:22.560196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:40683 (local address 127.0.0.1:59872)
0928 06:40:22.560381 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:22.560396 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:22.560468 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:22.560634 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:22.560635 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:22.560638 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0928 06:40:22.560723 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:22.560725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:22.561338 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:22.561339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:22.562286 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:22.562288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:22.562810 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:22.562812 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:22.562892 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:22.562915 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 06:40:22.583388 (+ 20473us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41935/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 06:40:22.583508 (+ 120us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.211.126:40683: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41935/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":84,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.18.211.126:40683: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41935/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 06:40:22.584537 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 24367
2026-09-28T06:40:22Z chronyd exiting
I20260928 06:40:22.619927 19277 test_util.cc:175] -----------------------------------------------
I20260928 06:40:22.620016 19277 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790577602522428-19277-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (228 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 06:40:22.620803 19277 mini_oidc.cc:152] Starting JWKS server
I20260928 06:40:22.621483 19277 webserver.cc:460] Webserver started at https://127.0.0.1:45625/ using document root <none> and password file <none>
I20260928 06:40:22.621528 19277 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 06:40:22.621667 19277 webserver.cc:460] Webserver started at http://127.0.0.1:45837/ using document root <none> and password file <none>
2026-09-28T06:40:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:40:22Z Disabled control of system clock
I20260928 06:40:22.628780 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:38715
--webserver_interface=127.18.211.126
--webserver_port=0
--builtin_ntp_servers=127.18.211.84:34087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:38715
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45625/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577602522428-19277-0/testchainca.cert with env {}
W20260928 06:40:22.705796 24470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:22.705996 24470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:22.706024 24470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:22.707170 24470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:40:22.707208 24470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:22.707232 24470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:22.707248 24470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:40:22.707264 24470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 06:40:22.707284 24470 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 06:40:22.707302 24470 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45625/jwks.json
I20260928 06:40:22.708794 24470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:34087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:38715
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:38715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45625/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577602522428-19277-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:22.709031 24470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:22.709218 24470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:22.713644 24478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:22.713671 24476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:22.713687 24475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:22.713943 24470 server_base.cc:1034] running on GCE node
I20260928 06:40:22.714150 24470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:22.714429 24470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:22.715581 24470 hybrid_clock.cc:648] HybridClock initialized: now 1790577622715554 us; error 41 us; skew 500 ppm
I20260928 06:40:22.716756 24470 webserver.cc:460] Webserver started at http://127.18.211.126:39605/ using document root <none> and password file <none>
I20260928 06:40:22.716964 24470 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:22.717023 24470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:22.717125 24470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:22.718163 24470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "4269d9ed65b049db96d8b6ec13c48d42"
format_stamp: "Formatted at 2026-09-28 06:40:22 on dist-test-slave-b28r"
server_key: "4cce9fd2cb9851df4141cdda6cf50409"
server_key_iv: "ce25571298d4c9ddc65f6e291c09c192"
server_key_version: "encryptionkey@0"
I20260928 06:40:22.718477 24470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "4269d9ed65b049db96d8b6ec13c48d42"
format_stamp: "Formatted at 2026-09-28 06:40:22 on dist-test-slave-b28r"
server_key: "4cce9fd2cb9851df4141cdda6cf50409"
server_key_iv: "ce25571298d4c9ddc65f6e291c09c192"
server_key_version: "encryptionkey@0"
I20260928 06:40:22.719693 24470 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:40:22.720412 24484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:22.720558 24470 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:40:22.720646 24470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "4269d9ed65b049db96d8b6ec13c48d42"
format_stamp: "Formatted at 2026-09-28 06:40:22 on dist-test-slave-b28r"
server_key: "4cce9fd2cb9851df4141cdda6cf50409"
server_key_iv: "ce25571298d4c9ddc65f6e291c09c192"
server_key_version: "encryptionkey@0"
I20260928 06:40:22.720702 24470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:22.739615 24470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:22.739909 24470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:22.740046 24470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:22.743736 24470 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:38715
I20260928 06:40:22.743780 24536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:38715 every 8 connection(s)
I20260928 06:40:22.744088 24470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:40:22.744750 24537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:22.747491 24537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4269d9ed65b049db96d8b6ec13c48d42: Bootstrap starting.
I20260928 06:40:22.748037 24537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4269d9ed65b049db96d8b6ec13c48d42: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:22.748291 24537 log.cc:824] T 00000000000000000000000000000000 P 4269d9ed65b049db96d8b6ec13c48d42: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:22.748982 24537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4269d9ed65b049db96d8b6ec13c48d42: No bootstrap required, opened a new log
I20260928 06:40:22.750563 24537 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4269d9ed65b049db96d8b6ec13c48d42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4269d9ed65b049db96d8b6ec13c48d42" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 38715 } }
I20260928 06:40:22.750698 24537 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4269d9ed65b049db96d8b6ec13c48d42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:22.750718 24537 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4269d9ed65b049db96d8b6ec13c48d42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4269d9ed65b049db96d8b6ec13c48d42, State: Initialized, Role: FOLLOWER
I20260928 06:40:22.750849 24537 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4269d9ed65b049db96d8b6ec13c48d42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4269d9ed65b049db96d8b6ec13c48d42" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 38715 } }
I20260928 06:40:22.750913 24537 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4269d9ed65b049db96d8b6ec13c48d42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:40:22.750932 24537 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4269d9ed65b049db96d8b6ec13c48d42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:40:22.750959 24537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4269d9ed65b049db96d8b6ec13c48d42 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:22.751510 24537 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4269d9ed65b049db96d8b6ec13c48d42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4269d9ed65b049db96d8b6ec13c48d42" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 38715 } }
I20260928 06:40:22.751601 24537 leader_election.cc:302] T 00000000000000000000000000000000 P 4269d9ed65b049db96d8b6ec13c48d42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4269d9ed65b049db96d8b6ec13c48d42; no voters:
I20260928 06:40:22.751770 24537 leader_election.cc:288] T 00000000000000000000000000000000 P 4269d9ed65b049db96d8b6ec13c48d42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:40:22.751813 24540 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4269d9ed65b049db96d8b6ec13c48d42 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:22.751992 24540 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4269d9ed65b049db96d8b6ec13c48d42 [term 1 LEADER]: Becoming Leader. State: Replica: 4269d9ed65b049db96d8b6ec13c48d42, State: Running, Role: LEADER
I20260928 06:40:22.752002 24537 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4269d9ed65b049db96d8b6ec13c48d42 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:40:22.752099 24540 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4269d9ed65b049db96d8b6ec13c48d42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4269d9ed65b049db96d8b6ec13c48d42" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 38715 } }
I20260928 06:40:22.752470 24542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4269d9ed65b049db96d8b6ec13c48d42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4269d9ed65b049db96d8b6ec13c48d42. Latest consensus state: current_term: 1 leader_uuid: "4269d9ed65b049db96d8b6ec13c48d42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4269d9ed65b049db96d8b6ec13c48d42" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 38715 } } }
I20260928 06:40:22.752595 24542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4269d9ed65b049db96d8b6ec13c48d42 [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:22.752650 24541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4269d9ed65b049db96d8b6ec13c48d42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4269d9ed65b049db96d8b6ec13c48d42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4269d9ed65b049db96d8b6ec13c48d42" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 38715 } } }
I20260928 06:40:22.752683 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 24470
I20260928 06:40:22.752739 24541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4269d9ed65b049db96d8b6ec13c48d42 [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:22.752791 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:40:22.752983 19277 external_mini_cluster.cc:1428] Setting key 66e4b5f8e1b27bf56b6be7f046df2e23
I20260928 06:40:22.753010 24549 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:40:22.753727 24549 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:40:22.755201 24549 catalog_manager.cc:1347] Generated new cluster ID: b8437470135048c19e435532bc4babf1
I20260928 06:40:22.755262 24549 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:40:22.757953 24559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:22.753590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39008 (local address 127.18.211.126:38715)
0928 06:40:22.753787 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:22.753796 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:22.753828 (+ 32us) server_negotiation.cc:407] Connection header received
0928 06:40:22.753943 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:22.753950 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:22.754019 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:22.754161 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:22.754398 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:22.755505 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:22.756247 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:22.756658 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:22.756775 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:22.757367 (+ 592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:22.757371 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:22.757372 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:40:22.757538 (+ 166us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:22.757573 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:40:22.757584 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:22.757637 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:22.757710 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 06:40:22.757794 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":10,"server-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260928 06:40:22.758741 19277 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:40:22.775017 24549 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:40:22.775727 24549 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:40:22.781941 24549 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4269d9ed65b049db96d8b6ec13c48d42: Generated new TSK 0
I20260928 06:40:22.782120 24549 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 06:40:22.814513 24567 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 06:40:22.814601 24567 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 06:40:22.814862 24559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:22.793318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39018 (local address 127.18.211.126:38715)
0928 06:40:22.793370 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:22.793372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:22.793468 (+ 96us) server_negotiation.cc:407] Connection header received
0928 06:40:22.793547 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:22.793548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:22.793561 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:22.793617 (+ 56us) server_negotiation.cc:226] Negotiated authn=JWT
0928 06:40:22.793737 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:22.794247 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:22.794974 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:22.795212 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:22.795266 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:22.795450 (+ 184us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 06:40:22.814639 (+ 19189us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45625/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 06:40:22.814808 (+ 169us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39018: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45625/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":27}
W20260928 06:40:22.814913 24559 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39018: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45625/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 06:40:22.814958 24566 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:22.793239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:38715 (local address 127.0.0.1:39018)
0928 06:40:22.793402 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:22.793412 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:22.793479 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:22.793620 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:22.793621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:22.793622 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0928 06:40:22.793679 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:22.793682 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:22.794324 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:22.794325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:22.794899 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:22.794901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:22.795288 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:22.795289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:22.795355 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:22.795360 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 06:40:22.814749 (+ 19389us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45625/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 06:40:22.814858 (+ 109us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.211.126:38715: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45625/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":91,"threads_started":1}
I20260928 06:40:22.815402 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 24470
2026-09-28T06:40:22Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (231 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 06:40:22.852471 19277 mini_oidc.cc:152] Starting JWKS server
I20260928 06:40:22.853121 19277 webserver.cc:460] Webserver started at https://127.0.0.1:38119/ using document root <none> and password file <none>
I20260928 06:40:22.853185 19277 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 06:40:22.853339 19277 webserver.cc:460] Webserver started at http://127.0.0.1:38031/ using document root <none> and password file <none>
2026-09-28T06:40:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:40:22Z Disabled control of system clock
I20260928 06:40:22.860236 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:42809
--webserver_interface=127.18.211.126
--webserver_port=0
--builtin_ntp_servers=127.18.211.84:32903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:42809
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38119/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 06:40:22.940338 24573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:22.940544 24573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:22.940572 24573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:22.941715 24573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:40:22.941753 24573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:22.941771 24573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:22.941787 24573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:40:22.941802 24573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 06:40:22.941819 24573 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 06:40:22.941841 24573 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38119/jwks.json
I20260928 06:40:22.943195 24573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:32903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:42809
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:42809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38119/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:22.943439 24573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:22.943694 24573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:22.948124 24579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:22.948256 24578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:22.948143 24581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:22.948427 24573 server_base.cc:1034] running on GCE node
I20260928 06:40:22.948638 24573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:22.948892 24573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:22.950031 24573 hybrid_clock.cc:648] HybridClock initialized: now 1790577622950017 us; error 29 us; skew 500 ppm
I20260928 06:40:22.951216 24573 webserver.cc:460] Webserver started at http://127.18.211.126:38635/ using document root <none> and password file <none>
I20260928 06:40:22.951455 24573 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:22.951505 24573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:22.951629 24573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:22.952641 24573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "75c8b02ef6af478980a80483470b504c"
format_stamp: "Formatted at 2026-09-28 06:40:22 on dist-test-slave-b28r"
server_key: "c258e6fd42f68009a38127dbdc58f741"
server_key_iv: "f4fff69ed181bc392fea67e231efeb65"
server_key_version: "encryptionkey@0"
I20260928 06:40:22.952960 24573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "75c8b02ef6af478980a80483470b504c"
format_stamp: "Formatted at 2026-09-28 06:40:22 on dist-test-slave-b28r"
server_key: "c258e6fd42f68009a38127dbdc58f741"
server_key_iv: "f4fff69ed181bc392fea67e231efeb65"
server_key_version: "encryptionkey@0"
I20260928 06:40:22.954146 24573 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:40:22.954761 24587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:22.954918 24573 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:40:22.954998 24573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "75c8b02ef6af478980a80483470b504c"
format_stamp: "Formatted at 2026-09-28 06:40:22 on dist-test-slave-b28r"
server_key: "c258e6fd42f68009a38127dbdc58f741"
server_key_iv: "f4fff69ed181bc392fea67e231efeb65"
server_key_version: "encryptionkey@0"
I20260928 06:40:22.955055 24573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:22.968340 24573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:22.968616 24573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:22.968737 24573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:22.972301 24573 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:42809
I20260928 06:40:22.972358 24639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:42809 every 8 connection(s)
I20260928 06:40:22.972642 24573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:40:22.973295 24640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:22.973877 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 24573
I20260928 06:40:22.973982 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:40:22.974151 19277 external_mini_cluster.cc:1428] Setting key e872ccd768dcaa2389ab0df1f672dd6b
I20260928 06:40:22.977058 24640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75c8b02ef6af478980a80483470b504c: Bootstrap starting.
I20260928 06:40:22.977634 24640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75c8b02ef6af478980a80483470b504c: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:22.977919 24640 log.cc:824] T 00000000000000000000000000000000 P 75c8b02ef6af478980a80483470b504c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:22.978649 24640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75c8b02ef6af478980a80483470b504c: No bootstrap required, opened a new log
I20260928 06:40:22.979789 24643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:22.974620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47072 (local address 127.18.211.126:42809)
0928 06:40:22.974825 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:22.974834 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:22.974868 (+ 34us) server_negotiation.cc:407] Connection header received
0928 06:40:22.975354 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:22.975364 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:22.975447 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:22.975599 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:22.975762 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:22.977065 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:22.977834 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:22.978351 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:22.978480 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:22.979107 (+ 627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:22.979112 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:22.979112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:40:22.979309 (+ 197us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:22.979349 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:40:22.979364 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:22.979428 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:22.979487 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 06:40:22.979562 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":126,"thread_start_us":74,"threads_started":1}
I20260928 06:40:22.980696 19277 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:40:22.980815 24640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75c8b02ef6af478980a80483470b504c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75c8b02ef6af478980a80483470b504c" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 42809 } }
I20260928 06:40:22.980955 24640 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75c8b02ef6af478980a80483470b504c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:22.980979 24640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75c8b02ef6af478980a80483470b504c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75c8b02ef6af478980a80483470b504c, State: Initialized, Role: FOLLOWER
I20260928 06:40:22.981094 24640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75c8b02ef6af478980a80483470b504c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75c8b02ef6af478980a80483470b504c" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 42809 } }
I20260928 06:40:22.981161 24640 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75c8b02ef6af478980a80483470b504c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:40:22.981184 24640 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75c8b02ef6af478980a80483470b504c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:40:22.981237 24640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75c8b02ef6af478980a80483470b504c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:22.981834 24640 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75c8b02ef6af478980a80483470b504c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75c8b02ef6af478980a80483470b504c" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 42809 } }
I20260928 06:40:22.981920 24640 leader_election.cc:302] T 00000000000000000000000000000000 P 75c8b02ef6af478980a80483470b504c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75c8b02ef6af478980a80483470b504c; no voters:
I20260928 06:40:22.982059 24640 leader_election.cc:288] T 00000000000000000000000000000000 P 75c8b02ef6af478980a80483470b504c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:40:22.982095 24645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75c8b02ef6af478980a80483470b504c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:22.982230 24645 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75c8b02ef6af478980a80483470b504c [term 1 LEADER]: Becoming Leader. State: Replica: 75c8b02ef6af478980a80483470b504c, State: Running, Role: LEADER
I20260928 06:40:22.982286 24640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75c8b02ef6af478980a80483470b504c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:40:22.982317 24645 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75c8b02ef6af478980a80483470b504c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75c8b02ef6af478980a80483470b504c" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 42809 } }
I20260928 06:40:22.982648 24646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75c8b02ef6af478980a80483470b504c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75c8b02ef6af478980a80483470b504c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75c8b02ef6af478980a80483470b504c" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 42809 } } }
I20260928 06:40:22.982652 24647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75c8b02ef6af478980a80483470b504c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75c8b02ef6af478980a80483470b504c. Latest consensus state: current_term: 1 leader_uuid: "75c8b02ef6af478980a80483470b504c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75c8b02ef6af478980a80483470b504c" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 42809 } } }
I20260928 06:40:22.982811 24647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75c8b02ef6af478980a80483470b504c [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:22.982765 24646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75c8b02ef6af478980a80483470b504c [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:22.983253 24653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:40:22.983757 24653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:40:22.985173 24653 catalog_manager.cc:1347] Generated new cluster ID: 5855e66ee8f541a4b853e0e72ce2bc6a
I20260928 06:40:22.985258 24653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:40:22.997745 24653 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:40:22.998260 24653 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:40:23.006201 24653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75c8b02ef6af478980a80483470b504c: Generated new TSK 0
I20260928 06:40:23.006438 24653 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:40:23.021391 24643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:23.017818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47086 (local address 127.18.211.126:42809)
0928 06:40:23.017872 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:23.017874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:23.017967 (+ 93us) server_negotiation.cc:407] Connection header received
0928 06:40:23.018041 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:23.018042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:23.018053 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:23.018111 (+ 58us) server_negotiation.cc:226] Negotiated authn=JWT
0928 06:40:23.018234 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:23.018759 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:23.019879 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:23.020077 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:23.020128 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:23.020295 (+ 167us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 06:40:23.021180 (+ 885us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38119/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 06:40:23.021319 (+ 139us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47086: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38119/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":29}
W20260928 06:40:23.021456 24643 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47086: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38119/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 06:40:23.021489 24669 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:23.017743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:42809 (local address 127.0.0.1:47086)
0928 06:40:23.017903 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:23.017914 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:23.017981 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:23.018122 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:23.018123 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:23.018125 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 06:40:23.018183 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:23.018185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:23.018827 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:23.018828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:23.019804 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:23.019807 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:23.020141 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:23.020142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:23.020197 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:23.020204 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 06:40:23.021275 (+ 1071us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38119/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 06:40:23.021374 (+ 99us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.211.126:42809: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38119/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":98,"threads_started":1}
I20260928 06:40:23.021710 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 24573
E20260928 06:40:23.032343 24670 webserver.cc:533] Webserver: sslize failed:
E20260928 06:40:23.032418 24670 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T06:40:23Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (205 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T06:40:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:40:23Z Disabled control of system clock
I20260928 06:40:23.065770 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:37921
--webserver_interface=127.18.211.126
--webserver_port=0
--builtin_ntp_servers=127.18.211.84:40637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:37921
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577602522428-19277-0/testchainca.cert with env {}
W20260928 06:40:23.144232 24674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:23.144433 24674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:23.144454 24674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:23.145542 24674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:40:23.145578 24674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:23.145591 24674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:23.145601 24674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:40:23.145624 24674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 06:40:23.145637 24674 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 06:40:23.145653 24674 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 06:40:23.147051 24674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:40637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:37921
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:37921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577602522428-19277-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:23.147281 24674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:23.147446 24674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:23.152026 24679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:23.152042 24680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:23.152042 24682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:23.152366 24674 server_base.cc:1034] running on GCE node
I20260928 06:40:23.152590 24674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:23.152853 24674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:23.154011 24674 hybrid_clock.cc:648] HybridClock initialized: now 1790577623153991 us; error 27 us; skew 500 ppm
I20260928 06:40:23.155429 24674 webserver.cc:460] Webserver started at http://127.18.211.126:45341/ using document root <none> and password file <none>
I20260928 06:40:23.155663 24674 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:23.155714 24674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:23.155827 24674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:23.156888 24674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "bb3f4f5f01554c1dae97afe0ee299788"
format_stamp: "Formatted at 2026-09-28 06:40:23 on dist-test-slave-b28r"
server_key: "4ec551f06122257725f724cdfe13de76"
server_key_iv: "b5c7379f0e86b0002b79ee461e5e0b7a"
server_key_version: "encryptionkey@0"
I20260928 06:40:23.157236 24674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "bb3f4f5f01554c1dae97afe0ee299788"
format_stamp: "Formatted at 2026-09-28 06:40:23 on dist-test-slave-b28r"
server_key: "4ec551f06122257725f724cdfe13de76"
server_key_iv: "b5c7379f0e86b0002b79ee461e5e0b7a"
server_key_version: "encryptionkey@0"
I20260928 06:40:23.158468 24674 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:40:23.159217 24688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:23.159389 24674 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:40:23.159473 24674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "bb3f4f5f01554c1dae97afe0ee299788"
format_stamp: "Formatted at 2026-09-28 06:40:23 on dist-test-slave-b28r"
server_key: "4ec551f06122257725f724cdfe13de76"
server_key_iv: "b5c7379f0e86b0002b79ee461e5e0b7a"
server_key_version: "encryptionkey@0"
I20260928 06:40:23.159541 24674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:23.166448 24674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:23.166703 24674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:23.166823 24674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:23.170567 24674 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:37921
I20260928 06:40:23.170619 24740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:37921 every 8 connection(s)
I20260928 06:40:23.170912 24674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:40:23.171887 24741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:23.174019 24741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb3f4f5f01554c1dae97afe0ee299788: Bootstrap starting.
I20260928 06:40:23.174526 24741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb3f4f5f01554c1dae97afe0ee299788: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:23.174750 24741 log.cc:824] T 00000000000000000000000000000000 P bb3f4f5f01554c1dae97afe0ee299788: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:23.175391 24741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb3f4f5f01554c1dae97afe0ee299788: No bootstrap required, opened a new log
I20260928 06:40:23.176919 24741 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb3f4f5f01554c1dae97afe0ee299788 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb3f4f5f01554c1dae97afe0ee299788" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 37921 } }
I20260928 06:40:23.177052 24741 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb3f4f5f01554c1dae97afe0ee299788 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:23.177073 24741 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb3f4f5f01554c1dae97afe0ee299788 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb3f4f5f01554c1dae97afe0ee299788, State: Initialized, Role: FOLLOWER
I20260928 06:40:23.177172 24741 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb3f4f5f01554c1dae97afe0ee299788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb3f4f5f01554c1dae97afe0ee299788" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 37921 } }
I20260928 06:40:23.177219 24741 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb3f4f5f01554c1dae97afe0ee299788 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:40:23.177238 24741 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb3f4f5f01554c1dae97afe0ee299788 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:40:23.177263 24741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb3f4f5f01554c1dae97afe0ee299788 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:23.177731 24741 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb3f4f5f01554c1dae97afe0ee299788 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb3f4f5f01554c1dae97afe0ee299788" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 37921 } }
I20260928 06:40:23.177810 24741 leader_election.cc:302] T 00000000000000000000000000000000 P bb3f4f5f01554c1dae97afe0ee299788 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb3f4f5f01554c1dae97afe0ee299788; no voters:
I20260928 06:40:23.177927 24741 leader_election.cc:288] T 00000000000000000000000000000000 P bb3f4f5f01554c1dae97afe0ee299788 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:40:23.178027 24744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb3f4f5f01554c1dae97afe0ee299788 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:23.178144 24741 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb3f4f5f01554c1dae97afe0ee299788 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:40:23.178179 24744 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb3f4f5f01554c1dae97afe0ee299788 [term 1 LEADER]: Becoming Leader. State: Replica: bb3f4f5f01554c1dae97afe0ee299788, State: Running, Role: LEADER
I20260928 06:40:23.178270 24744 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb3f4f5f01554c1dae97afe0ee299788 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb3f4f5f01554c1dae97afe0ee299788" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 37921 } }
I20260928 06:40:23.178725 24746 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb3f4f5f01554c1dae97afe0ee299788 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb3f4f5f01554c1dae97afe0ee299788. Latest consensus state: current_term: 1 leader_uuid: "bb3f4f5f01554c1dae97afe0ee299788" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb3f4f5f01554c1dae97afe0ee299788" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 37921 } } }
I20260928 06:40:23.178802 24746 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb3f4f5f01554c1dae97afe0ee299788 [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:23.179001 24753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:40:23.178958 24745 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb3f4f5f01554c1dae97afe0ee299788 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb3f4f5f01554c1dae97afe0ee299788" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb3f4f5f01554c1dae97afe0ee299788" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 37921 } } }
I20260928 06:40:23.179164 24745 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb3f4f5f01554c1dae97afe0ee299788 [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:23.179582 24753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:40:23.179816 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 24674
I20260928 06:40:23.179912 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:40:23.180076 19277 external_mini_cluster.cc:1428] Setting key 64ef7bda4b080f5d0fdd0ee7d439f45c
I20260928 06:40:23.181079 24753 catalog_manager.cc:1347] Generated new cluster ID: 63ce703d5369485f807b6bbd418e839f
I20260928 06:40:23.181128 24753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:40:23.187173 24753 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:40:23.187671 24753 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:40:23.188741 24763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:23.180460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43004 (local address 127.18.211.126:37921)
0928 06:40:23.180639 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:23.180648 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:23.183320 (+ 2672us) server_negotiation.cc:407] Connection header received
0928 06:40:23.183449 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:23.183457 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:23.183542 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:23.183701 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:23.183867 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:23.185271 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:23.186321 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:23.186963 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:23.187117 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:23.188010 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:23.188014 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:23.188015 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:40:23.188220 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:23.188259 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:40:23.188280 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:23.188339 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:23.188432 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 06:40:23.188526 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260928 06:40:23.189613 19277 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:40:23.192411 24753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb3f4f5f01554c1dae97afe0ee299788: Generated new TSK 0
I20260928 06:40:23.192562 24753 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:40:23.268517 24763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:23.202598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43014 (local address 127.18.211.126:37921)
0928 06:40:23.202641 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:23.202643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:23.202711 (+ 68us) server_negotiation.cc:407] Connection header received
0928 06:40:23.202761 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:23.202762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:23.202774 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:23.202853 (+ 79us) server_negotiation.cc:226] Negotiated authn=JWT
0928 06:40:23.202956 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:23.203725 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:23.204478 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:23.204777 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:23.204841 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:23.204955 (+ 114us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 06:40:23.268233 (+ 63278us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 06:40:23.268432 (+ 199us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43014: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":26}
W20260928 06:40:23.268594 24763 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43014: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 06:40:23.268630 24770 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:23.202520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:37921 (local address 127.0.0.1:43014)
0928 06:40:23.202647 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:23.202657 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:23.202711 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:23.202851 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:23.202851 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:23.202853 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0928 06:40:23.202904 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:23.202906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:23.203812 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:23.203812 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:23.204409 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:23.204411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:23.204840 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:23.204841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:23.204876 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:23.204880 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 06:40:23.268399 (+ 63519us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 06:40:23.268527 (+ 128us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.211.126:37921: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":75,"threads_started":1}
I20260928 06:40:23.268821 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 24674
2026-09-28T06:40:23Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (226 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (85 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (84 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:40:23 dist-test-slave-b28r krb5kdc[24782](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:40:23 dist-test-slave-b28r krb5kdc[24782](info): set up 2 sockets
Sep 28 06:40:23 dist-test-slave-b28r krb5kdc[24782](info): commencing operation
krb5kdc: starting...
W20260928 06:40:24.384130 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:40:24 dist-test-slave-b28r krb5kdc[24782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:40:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:40:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.211.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
I20260928 06:40:24.497527 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:44905
--webserver_interface=127.18.211.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:44905
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:24.576692 24795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:24.576892 24795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:24.576921 24795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:24.578039 24795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:40:24.578081 24795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:24.578100 24795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:24.578116 24795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:40:24.578133 24795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:40:24.579614 24795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:44905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:44905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:24.579849 24795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:24.580049 24795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:40:24.584781 24795 server_base.cc:1034] running on GCE node
W20260928 06:40:24.584754 24803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:24.584764 24800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:24.584784 24801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:24.585160 24795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:24.585505 24795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:24.586656 24795 hybrid_clock.cc:648] HybridClock initialized: now 1790577624586632 us; error 37 us; skew 500 ppm
Sep 28 06:40:24 dist-test-slave-b28r krb5kdc[24782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577624, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:24.589042 24795 init.cc:377] Logged in from keytab as kudu/127.18.211.126@KRBTEST.COM (short username kudu)
I20260928 06:40:24.589478 24795 webserver.cc:460] Webserver started at http://127.18.211.126:35527/ using document root <none> and password file <none>
I20260928 06:40:24.589691 24795 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:24.589740 24795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:24.589861 24795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:24.590884 24795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "bc0891713c4f4ed68e5c2e8f987d0c81"
format_stamp: "Formatted at 2026-09-28 06:40:24 on dist-test-slave-b28r"
server_key: "37806ba3106cd6bbaea9f90c85c3edf6"
server_key_iv: "06b5ab028c40724dec93086abbf16dc4"
server_key_version: "encryptionkey@0"
I20260928 06:40:24.591277 24795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "bc0891713c4f4ed68e5c2e8f987d0c81"
format_stamp: "Formatted at 2026-09-28 06:40:24 on dist-test-slave-b28r"
server_key: "37806ba3106cd6bbaea9f90c85c3edf6"
server_key_iv: "06b5ab028c40724dec93086abbf16dc4"
server_key_version: "encryptionkey@0"
I20260928 06:40:24.592504 24795 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:40:24.593181 24810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:24.593362 24795 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:40:24.593444 24795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "bc0891713c4f4ed68e5c2e8f987d0c81"
format_stamp: "Formatted at 2026-09-28 06:40:24 on dist-test-slave-b28r"
server_key: "37806ba3106cd6bbaea9f90c85c3edf6"
server_key_iv: "06b5ab028c40724dec93086abbf16dc4"
server_key_version: "encryptionkey@0"
I20260928 06:40:24.593509 24795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:24.616575 24795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:24.618773 24795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:24.618914 24795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:24.622783 24795 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:44905
I20260928 06:40:24.622828 24862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:44905 every 8 connection(s)
I20260928 06:40:24.623189 24795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:40:24.624042 24863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:24.626282 24863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc0891713c4f4ed68e5c2e8f987d0c81: Bootstrap starting.
I20260928 06:40:24.626822 24863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc0891713c4f4ed68e5c2e8f987d0c81: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:24.627094 24863 log.cc:824] T 00000000000000000000000000000000 P bc0891713c4f4ed68e5c2e8f987d0c81: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:24.627802 24863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc0891713c4f4ed68e5c2e8f987d0c81: No bootstrap required, opened a new log
I20260928 06:40:24.629379 24863 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc0891713c4f4ed68e5c2e8f987d0c81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0891713c4f4ed68e5c2e8f987d0c81" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44905 } }
I20260928 06:40:24.629515 24863 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc0891713c4f4ed68e5c2e8f987d0c81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:24.629536 24863 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc0891713c4f4ed68e5c2e8f987d0c81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc0891713c4f4ed68e5c2e8f987d0c81, State: Initialized, Role: FOLLOWER
I20260928 06:40:24.629664 24863 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc0891713c4f4ed68e5c2e8f987d0c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0891713c4f4ed68e5c2e8f987d0c81" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44905 } }
I20260928 06:40:24.629742 24863 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc0891713c4f4ed68e5c2e8f987d0c81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:40:24.629776 24863 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc0891713c4f4ed68e5c2e8f987d0c81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:40:24.629824 24863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc0891713c4f4ed68e5c2e8f987d0c81 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:24.630303 24863 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc0891713c4f4ed68e5c2e8f987d0c81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0891713c4f4ed68e5c2e8f987d0c81" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44905 } }
I20260928 06:40:24.630393 24863 leader_election.cc:302] T 00000000000000000000000000000000 P bc0891713c4f4ed68e5c2e8f987d0c81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc0891713c4f4ed68e5c2e8f987d0c81; no voters:
I20260928 06:40:24.630545 24863 leader_election.cc:288] T 00000000000000000000000000000000 P bc0891713c4f4ed68e5c2e8f987d0c81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:40:24.630594 24866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc0891713c4f4ed68e5c2e8f987d0c81 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:24.630796 24863 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc0891713c4f4ed68e5c2e8f987d0c81 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:40:24.630954 24866 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc0891713c4f4ed68e5c2e8f987d0c81 [term 1 LEADER]: Becoming Leader. State: Replica: bc0891713c4f4ed68e5c2e8f987d0c81, State: Running, Role: LEADER
I20260928 06:40:24.631048 24866 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc0891713c4f4ed68e5c2e8f987d0c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0891713c4f4ed68e5c2e8f987d0c81" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44905 } }
I20260928 06:40:24.631539 24867 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc0891713c4f4ed68e5c2e8f987d0c81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc0891713c4f4ed68e5c2e8f987d0c81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0891713c4f4ed68e5c2e8f987d0c81" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44905 } } }
I20260928 06:40:24.631553 24868 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc0891713c4f4ed68e5c2e8f987d0c81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc0891713c4f4ed68e5c2e8f987d0c81. Latest consensus state: current_term: 1 leader_uuid: "bc0891713c4f4ed68e5c2e8f987d0c81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc0891713c4f4ed68e5c2e8f987d0c81" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44905 } } }
I20260928 06:40:24.631634 24867 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc0891713c4f4ed68e5c2e8f987d0c81 [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:24.631650 24868 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc0891713c4f4ed68e5c2e8f987d0c81 [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:24.631878 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 24795
I20260928 06:40:24.631970 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:40:24.632133 19277 external_mini_cluster.cc:1428] Setting key 1daa41893a46fc918483d326afe9c7dc
W20260928 06:40:24.632612 24881 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc0891713c4f4ed68e5c2e8f987d0c81: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:40:24.632666 24881 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:40:24.632710 24875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:40:24.632853 24875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:40:24.634238 24875 catalog_manager.cc:1347] Generated new cluster ID: f4823fd1facd4fc2a56f9cd68b7c523c
I20260928 06:40:24.634289 24875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:40:24 dist-test-slave-b28r krb5kdc[24782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:24.644577 24884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:24.632726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36888 (local address 127.18.211.126:44905)
0928 06:40:24.633103 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:24.633114 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:24.633146 (+ 32us) server_negotiation.cc:407] Connection header received
0928 06:40:24.633349 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:24.633355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:24.633410 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:24.633548 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:24.634342 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:24.635788 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:24.636883 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:24.637522 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:24.637622 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:24.640534 (+ 2912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:24.640540 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:24.640540 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:24.640564 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:24.642475 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:24.642876 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:24.642880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:24.642880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:24.642998 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:24.643239 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:24.643240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:24.643240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:24.643567 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:24.643740 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:24.644178 (+ 438us) server_negotiation.cc:299] Negotiation successful
0928 06:40:24.644336 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":77,"threads_started":1}
I20260928 06:40:24.644954 24875 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:40:24.645650 24875 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:40:24.672663 24875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc0891713c4f4ed68e5c2e8f987d0c81: Generated new TSK 0
I20260928 06:40:24.672901 24875 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.211.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab.
Entry for principal kudu/127.18.211.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab.
I20260928 06:40:24.704169 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.65:0
--local_ip_for_outbound_sockets=127.18.211.65
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:44905
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:24.782490 24889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:24.782680 24889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:24.782701 24889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:24.783851 24889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:24.783890 24889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:24.783917 24889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.65
I20260928 06:40:24.785346 24889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:44905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.65
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:24.785554 24889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:24.785749 24889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:24.790292 24895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:24.790343 24894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:24.790343 24889 server_base.cc:1034] running on GCE node
W20260928 06:40:24.790294 24897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:24.790661 24889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:24.790927 24889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:24.792078 24889 hybrid_clock.cc:648] HybridClock initialized: now 1790577624792053 us; error 42 us; skew 500 ppm
Sep 28 06:40:24 dist-test-slave-b28r krb5kdc[24782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577624, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:24.794401 24889 init.cc:377] Logged in from keytab as kudu/127.18.211.65@KRBTEST.COM (short username kudu)
I20260928 06:40:24.794833 24889 webserver.cc:460] Webserver started at http://127.18.211.65:37167/ using document root <none> and password file <none>
I20260928 06:40:24.795028 24889 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:24.795069 24889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:24.795212 24889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:24.796250 24889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-0/data/instance:
uuid: "d36af94c446f437eaa7cf09bd8c2e77c"
format_stamp: "Formatted at 2026-09-28 06:40:24 on dist-test-slave-b28r"
server_key: "148023dc4e69af9de2e53eeb893cf089"
server_key_iv: "16f81c72f6115b14c858b04c45d68058"
server_key_version: "encryptionkey@0"
I20260928 06:40:24.796582 24889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance:
uuid: "d36af94c446f437eaa7cf09bd8c2e77c"
format_stamp: "Formatted at 2026-09-28 06:40:24 on dist-test-slave-b28r"
server_key: "148023dc4e69af9de2e53eeb893cf089"
server_key_iv: "16f81c72f6115b14c858b04c45d68058"
server_key_version: "encryptionkey@0"
I20260928 06:40:24.797863 24889 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:40:24.798527 24904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:24.798658 24889 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:40:24.798738 24889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-0/wal
uuid: "d36af94c446f437eaa7cf09bd8c2e77c"
format_stamp: "Formatted at 2026-09-28 06:40:24 on dist-test-slave-b28r"
server_key: "148023dc4e69af9de2e53eeb893cf089"
server_key_iv: "16f81c72f6115b14c858b04c45d68058"
server_key_version: "encryptionkey@0"
I20260928 06:40:24.798806 24889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:24.813058 24889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:24.814980 24889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:24.815150 24889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:24.815333 24889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:24.815563 24889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:24.815591 24889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:24.815610 24889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:24.815626 24889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:24.821060 24889 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.65:46541
I20260928 06:40:24.821102 25017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.65:46541 every 8 connection(s)
I20260928 06:40:24.821386 24889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:40:24 dist-test-slave-b28r krb5kdc[24782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577624, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:24.828991 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 24889
I20260928 06:40:24.829090 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance
I20260928 06:40:24.829288 19277 external_mini_cluster.cc:1428] Setting key 3eaa09f6644385b7c8cf14c1a316daa3
I20260928 06:40:24.830130 24884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:24.822047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:40655 (local address 127.18.211.126:44905)
0928 06:40:24.822091 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:24.822093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:24.822277 (+ 184us) server_negotiation.cc:407] Connection header received
0928 06:40:24.823230 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:24.823232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:24.823246 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:24.823306 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:24.823877 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:24.824378 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:24.825304 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:24.825507 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:24.825556 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:24.827524 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:24.827525 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:24.827526 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:24.827533 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:24.828718 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:24.829206 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:24.829207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:24.829208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:24.829264 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:24.829524 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:24.829525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:24.829526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:24.829651 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:24.829709 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:24.829986 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 06:40:24.830069 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:40:24.830366 25020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:24.822021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44905 (local address 127.18.211.65:40655)
0928 06:40:24.822162 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:24.822196 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:24.823109 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:24.823362 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:24.823365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:24.823592 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:24.823780 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:24.823784 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:24.824445 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:24.824445 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:24.825236 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:24.825238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:24.825568 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:24.825569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:24.825661 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:24.826115 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:24.826123 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:24.827441 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:24.828799 (+ 1358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:24.828807 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:24.828809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:24.829095 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:24.829324 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:24.829325 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:24.829325 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:24.829456 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:24.829712 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:24.829717 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:24.829867 (+ 150us) client_negotiation.cc:769] Sending connection context
0928 06:40:24.829959 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 06:40:24.830116 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":83,"threads_started":1}
I20260928 06:40:24.830885 25018 heartbeater.cc:347] Connected to a master server at 127.18.211.126:44905
I20260928 06:40:24.830984 25018 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:24.831187 25018 heartbeater.cc:511] Master 127.18.211.126:44905 requested a full tablet report, sending...
I20260928 06:40:24.831708 24827 ts_manager.cc:194] Registered new tserver with Master: d36af94c446f437eaa7cf09bd8c2e77c (127.18.211.65:46541)
I20260928 06:40:24.832463 24827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.65@KRBTEST.COM'} at 127.18.211.65:40655
WARNING: no policy specified for kudu/127.18.211.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab.
Entry for principal kudu/127.18.211.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab.
I20260928 06:40:24.851496 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.66:0
--local_ip_for_outbound_sockets=127.18.211.66
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:44905
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:24.929989 25023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:24.930197 25023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:24.930228 25023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:24.931483 25023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:24.931522 25023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:24.931556 25023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.66
I20260928 06:40:24.933015 25023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:44905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.66
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:24.933254 25023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:24.933451 25023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:24.938052 25031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:24.938033 25028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:24.938033 25029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:24.938392 25023 server_base.cc:1034] running on GCE node
I20260928 06:40:24.938611 25023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:24.938841 25023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:24.939973 25023 hybrid_clock.cc:648] HybridClock initialized: now 1790577624939957 us; error 31 us; skew 500 ppm
Sep 28 06:40:24 dist-test-slave-b28r krb5kdc[24782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577624, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:24.944892 25023 init.cc:377] Logged in from keytab as kudu/127.18.211.66@KRBTEST.COM (short username kudu)
I20260928 06:40:24.945338 25023 webserver.cc:460] Webserver started at http://127.18.211.66:46453/ using document root <none> and password file <none>
I20260928 06:40:24.945541 25023 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:24.945588 25023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:24.945705 25023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:24.946725 25023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-1/data/instance:
uuid: "456f30d77b49495e9d9db365e9d3e5f4"
format_stamp: "Formatted at 2026-09-28 06:40:24 on dist-test-slave-b28r"
server_key: "26d5944d5c8b77bab0b0cdbde325aee7"
server_key_iv: "6115ffcd599d5de98a5268248a4e1839"
server_key_version: "encryptionkey@0"
I20260928 06:40:24.947043 25023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance:
uuid: "456f30d77b49495e9d9db365e9d3e5f4"
format_stamp: "Formatted at 2026-09-28 06:40:24 on dist-test-slave-b28r"
server_key: "26d5944d5c8b77bab0b0cdbde325aee7"
server_key_iv: "6115ffcd599d5de98a5268248a4e1839"
server_key_version: "encryptionkey@0"
I20260928 06:40:24.948271 25023 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:24.948983 25038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:24.949146 25023 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:40:24.949223 25023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-1/wal
uuid: "456f30d77b49495e9d9db365e9d3e5f4"
format_stamp: "Formatted at 2026-09-28 06:40:24 on dist-test-slave-b28r"
server_key: "26d5944d5c8b77bab0b0cdbde325aee7"
server_key_iv: "6115ffcd599d5de98a5268248a4e1839"
server_key_version: "encryptionkey@0"
I20260928 06:40:24.949282 25023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:24.964573 25023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:24.966758 25023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:24.966913 25023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:24.967244 25023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:24.967588 25023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:24.967626 25023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:24.967659 25023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:24.967687 25023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:24.974030 25023 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.66:34267
I20260928 06:40:24.974071 25151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.66:34267 every 8 connection(s)
I20260928 06:40:24.974400 25023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
I20260928 06:40:24.975675 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 25023
I20260928 06:40:24.975775 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance
I20260928 06:40:24.975956 19277 external_mini_cluster.cc:1428] Setting key 0cffbe6776a15d909a9ae797c90f84cd
Sep 28 06:40:24 dist-test-slave-b28r krb5kdc[24782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577624, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.211.67@KRBTEST.COM; defaulting to no policy
I20260928 06:40:24.984663 24884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:24.975131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:46575 (local address 127.18.211.126:44905)
0928 06:40:24.975194 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:24.975196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:24.975385 (+ 189us) server_negotiation.cc:407] Connection header received
0928 06:40:24.976579 (+ 1194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:24.976580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:24.976592 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:24.976668 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:24.977317 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:24.978022 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:24.979459 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:24.979663 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:24.979725 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:24.982079 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:24.982081 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:24.982081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:24.982089 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:24.983384 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:24.983745 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:24.983746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:24.983747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:24.983797 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:24.984031 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:24.984032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:24.984032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:24.984142 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:24.984219 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:24.984472 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 06:40:24.984557 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 06:40:24.984812 25154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:24.975068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44905 (local address 127.18.211.66:46575)
0928 06:40:24.975266 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:24.975303 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:24.976463 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:24.976766 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:24.976771 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:24.977034 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:24.977221 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:24.977225 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:24.978120 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:24.978121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:24.979353 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:24.979356 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:24.979738 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:24.979740 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:24.979888 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:24.980450 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:24.980459 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:24.981986 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:24.983456 (+ 1470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:24.983463 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:24.983465 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:24.983680 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:24.983851 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:24.983852 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:24.983852 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:24.983973 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:24.984201 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:24.984207 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:24.984352 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 06:40:24.984443 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 06:40:24.984599 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":95,"threads_started":1}
I20260928 06:40:24.985327 25152 heartbeater.cc:347] Connected to a master server at 127.18.211.126:44905
I20260928 06:40:24.985438 25152 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:24.985704 25152 heartbeater.cc:511] Master 127.18.211.126:44905 requested a full tablet report, sending...
I20260928 06:40:24.986125 24827 ts_manager.cc:194] Registered new tserver with Master: 456f30d77b49495e9d9db365e9d3e5f4 (127.18.211.66:34267)
I20260928 06:40:24.986508 24827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.66@KRBTEST.COM'} at 127.18.211.66:46575
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab.
Entry for principal kudu/127.18.211.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab.
I20260928 06:40:25.001309 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.67:0
--local_ip_for_outbound_sockets=127.18.211.67
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:44905
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:36391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:25.080665 25157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:25.080873 25157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:25.080894 25157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:25.081998 25157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:25.082031 25157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:25.082055 25157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.67
I20260928 06:40:25.083524 25157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:36391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:44905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25157
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.67
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:25.083712 25157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:25.083904 25157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:25.088709 25165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:25.088733 25163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:25.088693 25162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:25.088871 25157 server_base.cc:1034] running on GCE node
I20260928 06:40:25.089143 25157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:25.089382 25157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:25.090524 25157 hybrid_clock.cc:648] HybridClock initialized: now 1790577625090505 us; error 35 us; skew 500 ppm
Sep 28 06:40:25 dist-test-slave-b28r krb5kdc[24782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577625, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:25.093080 25157 init.cc:377] Logged in from keytab as kudu/127.18.211.67@KRBTEST.COM (short username kudu)
I20260928 06:40:25.093535 25157 webserver.cc:460] Webserver started at http://127.18.211.67:37759/ using document root <none> and password file <none>
I20260928 06:40:25.093730 25157 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:25.093792 25157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:25.093922 25157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:25.095028 25157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-2/data/instance:
uuid: "8994d6cb41a747128e31745781bb7524"
format_stamp: "Formatted at 2026-09-28 06:40:25 on dist-test-slave-b28r"
server_key: "e216d24e8a63ea19c471f6917adb6029"
server_key_iv: "f7acea64990b11fd056f88561cc62ba9"
server_key_version: "encryptionkey@0"
I20260928 06:40:25.095388 25157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance:
uuid: "8994d6cb41a747128e31745781bb7524"
format_stamp: "Formatted at 2026-09-28 06:40:25 on dist-test-slave-b28r"
server_key: "e216d24e8a63ea19c471f6917adb6029"
server_key_iv: "f7acea64990b11fd056f88561cc62ba9"
server_key_version: "encryptionkey@0"
I20260928 06:40:25.096652 25157 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:25.097513 25172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:25.097662 25157 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:25.097739 25157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-2/wal
uuid: "8994d6cb41a747128e31745781bb7524"
format_stamp: "Formatted at 2026-09-28 06:40:25 on dist-test-slave-b28r"
server_key: "e216d24e8a63ea19c471f6917adb6029"
server_key_iv: "f7acea64990b11fd056f88561cc62ba9"
server_key_version: "encryptionkey@0"
I20260928 06:40:25.097811 25157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:25.110322 25157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:25.112228 25157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:25.112356 25157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:25.112594 25157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:25.112905 25157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:25.112937 25157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:25.112967 25157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:25.112990 25157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:25.118599 25157 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.67:44359
I20260928 06:40:25.118657 25285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.67:44359 every 8 connection(s)
I20260928 06:40:25.118963 25157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:40:25 dist-test-slave-b28r krb5kdc[24782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577625, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:25.125977 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 25157
I20260928 06:40:25.126081 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance
I20260928 06:40:25.126284 19277 external_mini_cluster.cc:1428] Setting key c83cf864a049c033ee5bdcbb50f14a03
I20260928 06:40:25.128487 24884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:25.119736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:54527 (local address 127.18.211.126:44905)
0928 06:40:25.119834 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:25.119837 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:25.120023 (+ 186us) server_negotiation.cc:407] Connection header received
0928 06:40:25.121027 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:25.121029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:25.121042 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:25.121095 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:25.121759 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:25.122251 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:25.123267 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:25.123458 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:25.123516 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:25.125712 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:25.125713 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:25.125713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:25.125720 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:25.127299 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:25.127665 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:25.127665 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:25.127666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:25.127727 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:25.127935 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:25.127936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:25.127936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:25.128039 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:25.128083 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:25.128335 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 06:40:25.128419 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 06:40:25.128551 25288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:25.119694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44905 (local address 127.18.211.67:54527)
0928 06:40:25.119891 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:25.119934 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:25.120905 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:25.121186 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:25.121190 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:25.121455 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:25.121658 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:25.121662 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:25.122333 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:25.122334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:25.123169 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:25.123172 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:25.123528 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:25.123529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:25.123649 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:25.124119 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:25.124127 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:25.125613 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:25.127358 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:25.127364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:25.127366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:25.127607 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:25.127787 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:25.127787 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:25.127788 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:25.127880 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:25.128076 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:25.128078 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:25.128190 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 06:40:25.128268 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 06:40:25.128388 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":79,"threads_started":1}
I20260928 06:40:25.128973 25286 heartbeater.cc:347] Connected to a master server at 127.18.211.126:44905
I20260928 06:40:25.129076 25286 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:25.129307 25286 heartbeater.cc:511] Master 127.18.211.126:44905 requested a full tablet report, sending...
I20260928 06:40:25.129665 24827 ts_manager.cc:194] Registered new tserver with Master: 8994d6cb41a747128e31745781bb7524 (127.18.211.67:44359)
I20260928 06:40:25.130036 24827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.67@KRBTEST.COM'} at 127.18.211.67:54527
I20260928 06:40:25.138098 19277 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/krb5kdc/test-admin.keytab.
Sep 28 06:40:25 dist-test-slave-b28r krb5kdc[24782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:25.152873 19277 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 06:40:25.154420 25293 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:25.153712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50268 (local address 127.18.211.65:46541)
0928 06:40:25.153885 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:25.153890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:25.153901 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:40:25.153926 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:25.153927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:25.153941 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:25.154005 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:25.154351 (+ 346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50268: BlockingRecv error: recv got EOF from 127.0.0.1:50268 (error 108)
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":93,"threads_started":1}
W20260928 06:40:25.154567 19277 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/krb5kdc/krb5cc' of type FILE
Sep 28 06:40:25 dist-test-slave-b28r krb5kdc[24782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:25.155511 19277 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 06:40:25 dist-test-slave-b28r krb5kdc[24782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:25.156354 19277 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 06:40:25 dist-test-slave-b28r krb5kdc[24782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.65@KRBTEST.COM
I20260928 06:40:25.163796 25293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:25.156866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50276 (local address 127.18.211.65:46541)
0928 06:40:25.156916 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:25.156918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:25.156928 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:40:25.157001 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:25.157002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:25.157012 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:25.157057 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:25.157462 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:25.158180 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:25.158899 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:25.159423 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:25.159498 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:25.161501 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:25.161509 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:25.161509 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:25.161515 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:25.162832 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:25.163136 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:25.163139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:25.163140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:25.163193 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:25.163359 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:25.163360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:25.163360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:25.163471 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:25.163561 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:25.163614 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 06:40:25.163673 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260928 06:40:25.165498 25293 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:25.165030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50292 (local address 127.18.211.65:46541)
0928 06:40:25.165085 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:25.165087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:25.165095 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:40:25.165149 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:25.165150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:25.165160 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:25.165203 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:25.165452 (+ 249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50292: BlockingRecv error: recv got EOF from 127.0.0.1:50292 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 06:40:25.165580 19277 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/krb5kdc/krb5cc' of type FILE
Sep 28 06:40:25 dist-test-slave-b28r krb5kdc[24782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:25.166564 19277 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 06:40:25 dist-test-slave-b28r krb5kdc[24782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:25.167409 19277 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 06:40:25 dist-test-slave-b28r krb5kdc[24782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.65@KRBTEST.COM
I20260928 06:40:25.173878 25293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:25.167901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50296 (local address 127.18.211.65:46541)
0928 06:40:25.167964 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:25.167965 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:25.167973 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:40:25.168016 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:25.168016 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:25.168024 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:25.168065 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:25.168375 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:25.168883 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:25.169506 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:25.169701 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:25.169769 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:25.171561 (+ 1792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:25.171562 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:25.171562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:25.171568 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:25.172903 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:25.173187 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:25.173188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:25.173188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:25.173245 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:25.173442 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:25.173443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:25.173443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:25.173537 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:25.173617 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:25.173716 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 06:40:25.173789 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 06:40:25.175339 25293 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:25.174812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50312 (local address 127.18.211.65:46541)
0928 06:40:25.174867 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:25.174869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:25.174912 (+ 43us) server_negotiation.cc:407] Connection header received
0928 06:40:25.174967 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:25.174968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:25.174977 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:25.175019 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:25.175295 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50312: BlockingRecv error: recv got EOF from 127.0.0.1:50312 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260928 06:40:25.175441 19277 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790577602522428-19277-0/krb5kdc/krb5cc' of type FILE
Sep 28 06:40:25 dist-test-slave-b28r krb5kdc[24782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:25.176298 19277 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 06:40:25 dist-test-slave-b28r krb5kdc[24782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:25.177023 19277 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 06:40:25 dist-test-slave-b28r krb5kdc[24782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577625, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.65@KRBTEST.COM
I20260928 06:40:25.183473 25293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:25.177502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50318 (local address 127.18.211.65:46541)
0928 06:40:25.177536 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:25.177538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:25.177604 (+ 66us) server_negotiation.cc:407] Connection header received
0928 06:40:25.177648 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:25.177648 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:25.177655 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:25.177698 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:25.178015 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:25.178449 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:25.179208 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:25.179398 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:25.179449 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:25.181344 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:25.181345 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:25.181345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:25.181351 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:25.182510 (+ 1159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:25.182819 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:25.182820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:25.182820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:25.182870 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:25.183073 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:25.183074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:25.183074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:25.183197 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:25.183293 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:25.183363 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 06:40:25.183415 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:40:25.184021 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 24889
I20260928 06:40:25.188479 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 25023
I20260928 06:40:25.192387 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 25157
I20260928 06:40:25.196151 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 24795
2026-09-28T06:40:25Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1758 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:40:25 dist-test-slave-b28r krb5kdc[25306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:40:25 dist-test-slave-b28r krb5kdc[25306](info): set up 2 sockets
Sep 28 06:40:25 dist-test-slave-b28r krb5kdc[25306](info): commencing operation
krb5kdc: starting...
W20260928 06:40:26.140029 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:40:26 dist-test-slave-b28r krb5kdc[25306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:40:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:40:26Z Disabled control of system clock
WARNING: no policy specified for oryx/127.18.211.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.211.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.211.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.126.keytab.
Entry for principal oryx/127.18.211.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.126.keytab.
I20260928 06:40:26.247810 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:44085
--webserver_interface=127.18.211.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.126.keytab
--principal=oryx/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:44085
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:26.326071 25319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:26.326277 25319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:26.326298 25319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:26.327508 25319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:40:26.327551 25319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:26.327564 25319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:26.327576 25319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:40:26.327589 25319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:40:26.329092 25319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:44085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.126.keytab
--principal=oryx/127.18.211.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:44085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:26.329324 25319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:26.329540 25319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:26.333993 25327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:26.334121 25325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:26.334010 25324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:26.334553 25319 server_base.cc:1034] running on GCE node
I20260928 06:40:26.334780 25319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:26.335055 25319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:26.336225 25319 hybrid_clock.cc:648] HybridClock initialized: now 1790577626336196 us; error 49 us; skew 500 ppm
Sep 28 06:40:26 dist-test-slave-b28r krb5kdc[25306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577626, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:26.338506 25319 init.cc:377] Logged in from keytab as oryx/127.18.211.126@KRBTEST.COM (short username oryx)
I20260928 06:40:26.338944 25319 webserver.cc:460] Webserver started at http://127.18.211.126:45205/ using document root <none> and password file <none>
I20260928 06:40:26.339176 25319 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:26.339229 25319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:26.339356 25319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:26.340370 25319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "f5d90cb26f1a4900a3e9a927bb0d79be"
format_stamp: "Formatted at 2026-09-28 06:40:26 on dist-test-slave-b28r"
server_key: "03f803bfa2f5fc092ec5b59c1006e484"
server_key_iv: "490f2f2a883c7981d860410c6e0b7888"
server_key_version: "encryptionkey@0"
I20260928 06:40:26.340714 25319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "f5d90cb26f1a4900a3e9a927bb0d79be"
format_stamp: "Formatted at 2026-09-28 06:40:26 on dist-test-slave-b28r"
server_key: "03f803bfa2f5fc092ec5b59c1006e484"
server_key_iv: "490f2f2a883c7981d860410c6e0b7888"
server_key_version: "encryptionkey@0"
I20260928 06:40:26.341946 25319 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:40:26.342630 25334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:26.342787 25319 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:40:26.342869 25319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "f5d90cb26f1a4900a3e9a927bb0d79be"
format_stamp: "Formatted at 2026-09-28 06:40:26 on dist-test-slave-b28r"
server_key: "03f803bfa2f5fc092ec5b59c1006e484"
server_key_iv: "490f2f2a883c7981d860410c6e0b7888"
server_key_version: "encryptionkey@0"
I20260928 06:40:26.342936 25319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:26.356735 25319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:26.358465 25319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:26.358595 25319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:26.362589 25319 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:44085
I20260928 06:40:26.362625 25386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:44085 every 8 connection(s)
I20260928 06:40:26.362951 25319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:40:26.363907 25387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:26.366039 25387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be: Bootstrap starting.
I20260928 06:40:26.366554 25387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:26.366786 25387 log.cc:824] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:26.367445 25387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be: No bootstrap required, opened a new log
I20260928 06:40:26.368973 25387 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } }
I20260928 06:40:26.369107 25387 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:26.369128 25387 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5d90cb26f1a4900a3e9a927bb0d79be, State: Initialized, Role: FOLLOWER
I20260928 06:40:26.369243 25387 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } }
I20260928 06:40:26.369321 25387 raft_consensus.cc:388] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:40:26.369349 25387 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:40:26.369377 25387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:26.370085 25387 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } }
I20260928 06:40:26.370172 25387 leader_election.cc:302] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f5d90cb26f1a4900a3e9a927bb0d79be; no voters:
I20260928 06:40:26.370292 25387 leader_election.cc:288] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:40:26.370347 25390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:26.370527 25390 raft_consensus.cc:686] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [term 1 LEADER]: Becoming Leader. State: Replica: f5d90cb26f1a4900a3e9a927bb0d79be, State: Running, Role: LEADER
I20260928 06:40:26.370525 25387 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:40:26.370651 25390 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } }
I20260928 06:40:26.371001 25392 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5d90cb26f1a4900a3e9a927bb0d79be. Latest consensus state: current_term: 1 leader_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } }
I20260928 06:40:26.371112 25392 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:26.371019 25391 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } }
I20260928 06:40:26.371202 25391 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:26.371369 25399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:40:26.371914 25399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:40:26.371969 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 25319
I20260928 06:40:26.372089 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:40:26.372253 19277 external_mini_cluster.cc:1428] Setting key 29d2299588dfd62304ef9fb63a2cceae
I20260928 06:40:26.373603 25399 catalog_manager.cc:1347] Generated new cluster ID: dce0b6ad27964fd8a9550617d17fad5b
I20260928 06:40:26.373653 25399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:40:26 dist-test-slave-b28r krb5kdc[25306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM
I20260928 06:40:26.383401 25409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:26.372858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43426 (local address 127.18.211.126:44085)
0928 06:40:26.373040 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:26.373050 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:26.373082 (+ 32us) server_negotiation.cc:407] Connection header received
0928 06:40:26.373213 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:26.373222 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:26.373280 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:26.373416 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:26.374212 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:26.375638 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:26.376668 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:26.377233 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:26.377318 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:26.379975 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:26.379978 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:26.379979 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:26.379991 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:26.381870 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:26.382219 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:26.382222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:26.382222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:26.382327 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:26.382492 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:26.382493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:26.382493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:26.382790 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:26.382942 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:26.383016 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 06:40:26.383175 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":74,"threads_started":1}
I20260928 06:40:26.394047 25399 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:40:26.394733 25399 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:40:26.410179 25399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be: Generated new TSK 0
I20260928 06:40:26.410414 25399 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.18.211.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.211.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.211.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.65.keytab.
Entry for principal oryx/127.18.211.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.65.keytab.
I20260928 06:40:26.445127 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.65:0
--local_ip_for_outbound_sockets=127.18.211.65
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:44085
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.65.keytab
--principal=oryx/127.18.211.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:26.524008 25413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:26.524194 25413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:26.524215 25413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:26.525326 25413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:26.525365 25413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:26.525390 25413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.65
I20260928 06:40:26.526790 25413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.65.keytab
--principal=oryx/127.18.211.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:44085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.65
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:26.527004 25413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:26.527235 25413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:26.531917 25421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:26.531956 25418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:26.531947 25419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:26.531985 25413 server_base.cc:1034] running on GCE node
I20260928 06:40:26.532331 25413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:26.532567 25413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:26.533712 25413 hybrid_clock.cc:648] HybridClock initialized: now 1790577626533701 us; error 27 us; skew 500 ppm
Sep 28 06:40:26 dist-test-slave-b28r krb5kdc[25306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577626, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:26.536062 25413 init.cc:377] Logged in from keytab as oryx/127.18.211.65@KRBTEST.COM (short username oryx)
I20260928 06:40:26.536509 25413 webserver.cc:460] Webserver started at http://127.18.211.65:44001/ using document root <none> and password file <none>
I20260928 06:40:26.536715 25413 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:26.536762 25413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:26.536881 25413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:26.537874 25413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-0/data/instance:
uuid: "538426449b59422daa1419fc97acf5a6"
format_stamp: "Formatted at 2026-09-28 06:40:26 on dist-test-slave-b28r"
server_key: "d689cfc63c3d15c336a039dac7b5c945"
server_key_iv: "c7fd1c9ba107bb235a3d0df2e091ca45"
server_key_version: "encryptionkey@0"
I20260928 06:40:26.538185 25413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance:
uuid: "538426449b59422daa1419fc97acf5a6"
format_stamp: "Formatted at 2026-09-28 06:40:26 on dist-test-slave-b28r"
server_key: "d689cfc63c3d15c336a039dac7b5c945"
server_key_iv: "c7fd1c9ba107bb235a3d0df2e091ca45"
server_key_version: "encryptionkey@0"
I20260928 06:40:26.539471 25413 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:40:26.540117 25428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:26.540294 25413 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:26.540371 25413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-0/wal
uuid: "538426449b59422daa1419fc97acf5a6"
format_stamp: "Formatted at 2026-09-28 06:40:26 on dist-test-slave-b28r"
server_key: "d689cfc63c3d15c336a039dac7b5c945"
server_key_iv: "c7fd1c9ba107bb235a3d0df2e091ca45"
server_key_version: "encryptionkey@0"
I20260928 06:40:26.540433 25413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:26.565711 25413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:26.567603 25413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:26.567732 25413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:26.567986 25413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:26.568333 25413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:26.568369 25413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:26.568392 25413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:26.568414 25413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:26.574081 25413 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.65:35203
I20260928 06:40:26.574138 25541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.65:35203 every 8 connection(s)
I20260928 06:40:26.574525 25413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
I20260928 06:40:26.579824 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 25413
I20260928 06:40:26.579917 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance
I20260928 06:40:26.580089 19277 external_mini_cluster.cc:1428] Setting key fca3e5ec16173fe91c8a13f0ed9fe36f
Sep 28 06:40:26 dist-test-slave-b28r krb5kdc[25306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577626, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM
I20260928 06:40:26.583529 25409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:26.575232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:52605 (local address 127.18.211.126:44085)
0928 06:40:26.575302 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:26.575305 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:26.575544 (+ 239us) server_negotiation.cc:407] Connection header received
0928 06:40:26.576008 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:26.576010 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:26.576025 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:26.576073 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:26.576948 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:26.577460 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:26.578377 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:26.578573 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:26.578624 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:26.580750 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:26.580751 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:26.580751 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:26.580757 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:26.582114 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:26.582569 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:26.582569 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:26.582570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:26.582619 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:26.582858 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:26.582859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:26.582859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:26.582952 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:26.582997 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:26.583358 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 06:40:26.583441 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 06:40:26.583581 25544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:26.575256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44085 (local address 127.18.211.65:52605)
0928 06:40:26.575424 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:26.575461 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:26.575931 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:26.576129 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:26.576132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:26.576673 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:26.576872 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:26.576878 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:26.577523 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:26.577523 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:26.578305 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:26.578307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:26.578630 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:26.578631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:26.578729 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:26.579219 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:26.579227 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:26.580655 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:26.582200 (+ 1545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:26.582207 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:26.582210 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:26.582497 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:26.582667 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:26.582668 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:26.582668 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:26.582804 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:26.583003 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:26.583010 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:26.583196 (+ 186us) client_negotiation.cc:769] Sending connection context
0928 06:40:26.583296 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 06:40:26.583423 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":90,"threads_started":1}
I20260928 06:40:26.584141 25542 heartbeater.cc:347] Connected to a master server at 127.18.211.126:44085
I20260928 06:40:26.584244 25542 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:26.584440 25542 heartbeater.cc:511] Master 127.18.211.126:44085 requested a full tablet report, sending...
I20260928 06:40:26.584935 25351 ts_manager.cc:194] Registered new tserver with Master: 538426449b59422daa1419fc97acf5a6 (127.18.211.65:35203)
I20260928 06:40:26.585500 25351 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.211.65@KRBTEST.COM'} at 127.18.211.65:52605
WARNING: no policy specified for oryx/127.18.211.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.211.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.211.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.66.keytab.
Entry for principal oryx/127.18.211.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.66.keytab.
I20260928 06:40:26.602982 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.66:0
--local_ip_for_outbound_sockets=127.18.211.66
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:44085
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.66.keytab
--principal=oryx/127.18.211.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:26.683640 25547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:26.683830 25547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:26.683852 25547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:26.684932 25547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:26.684967 25547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:26.684990 25547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.66
I20260928 06:40:26.686354 25547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.66.keytab
--principal=oryx/127.18.211.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:44085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.66
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:26.686528 25547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:26.686703 25547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:26.691419 25553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:26.691437 25552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:26.691565 25547 server_base.cc:1034] running on GCE node
W20260928 06:40:26.691431 25555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:26.691810 25547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:26.692045 25547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:26.693173 25547 hybrid_clock.cc:648] HybridClock initialized: now 1790577626693163 us; error 27 us; skew 500 ppm
Sep 28 06:40:26 dist-test-slave-b28r krb5kdc[25306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577626, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:26.695546 25547 init.cc:377] Logged in from keytab as oryx/127.18.211.66@KRBTEST.COM (short username oryx)
I20260928 06:40:26.695951 25547 webserver.cc:460] Webserver started at http://127.18.211.66:34153/ using document root <none> and password file <none>
I20260928 06:40:26.696122 25547 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:26.696159 25547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:26.696249 25547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:26.697257 25547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-1/data/instance:
uuid: "becf12ef2f814e66b83a1ddee4fa138b"
format_stamp: "Formatted at 2026-09-28 06:40:26 on dist-test-slave-b28r"
server_key: "dc4e69dfbffd56d3fee2206e8d44609b"
server_key_iv: "058b46256bf72a7cb2ce458c817628e0"
server_key_version: "encryptionkey@0"
I20260928 06:40:26.697570 25547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance:
uuid: "becf12ef2f814e66b83a1ddee4fa138b"
format_stamp: "Formatted at 2026-09-28 06:40:26 on dist-test-slave-b28r"
server_key: "dc4e69dfbffd56d3fee2206e8d44609b"
server_key_iv: "058b46256bf72a7cb2ce458c817628e0"
server_key_version: "encryptionkey@0"
I20260928 06:40:26.698736 25547 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:40:26.699415 25562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:26.699589 25547 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:26.699674 25547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-1/wal
uuid: "becf12ef2f814e66b83a1ddee4fa138b"
format_stamp: "Formatted at 2026-09-28 06:40:26 on dist-test-slave-b28r"
server_key: "dc4e69dfbffd56d3fee2206e8d44609b"
server_key_iv: "058b46256bf72a7cb2ce458c817628e0"
server_key_version: "encryptionkey@0"
I20260928 06:40:26.699738 25547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:26.709398 25547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:26.711491 25547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:26.711620 25547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:26.711836 25547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:26.712119 25547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:26.712152 25547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:26.712172 25547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:26.712196 25547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:26.717442 25547 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.66:44887
I20260928 06:40:26.717514 25675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.66:44887 every 8 connection(s)
I20260928 06:40:26.717772 25547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:40:26 dist-test-slave-b28r krb5kdc[25306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577626, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM
I20260928 06:40:26.726667 25409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:26.718539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:54499 (local address 127.18.211.126:44085)
0928 06:40:26.718599 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:26.718601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:26.718780 (+ 179us) server_negotiation.cc:407] Connection header received
0928 06:40:26.719302 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:26.719303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:26.719315 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:26.719371 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:26.720322 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:26.720847 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:26.721887 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:26.722083 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:26.722139 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:26.724306 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:26.724308 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:26.724308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:26.724315 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:26.725599 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:26.725967 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:26.725968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:26.725968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:26.726017 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:26.726195 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:26.726196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:26.726196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:26.726292 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:26.726336 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:26.726542 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 06:40:26.726603 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 06:40:26.726861 25678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:26.718468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44085 (local address 127.18.211.66:54499)
0928 06:40:26.718651 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:26.718692 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:26.719214 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:26.719437 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:26.719440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:26.720002 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:26.720189 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:26.720194 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:26.720951 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:26.720953 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:26.721795 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:26.721797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:26.722166 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:26.722168 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:26.722286 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:26.722759 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:26.722767 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:26.724177 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:26.725679 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:26.725686 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:26.725688 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:26.725908 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:26.726065 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:26.726066 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:26.726066 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:26.726154 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:26.726333 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:26.726336 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:26.726448 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 06:40:26.726527 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 06:40:26.726643 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":92,"threads_started":1}
I20260928 06:40:26.727370 25676 heartbeater.cc:347] Connected to a master server at 127.18.211.126:44085
I20260928 06:40:26.727459 25676 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:26.727655 25676 heartbeater.cc:511] Master 127.18.211.126:44085 requested a full tablet report, sending...
I20260928 06:40:26.727655 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 25547
I20260928 06:40:26.727749 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance
I20260928 06:40:26.727959 19277 external_mini_cluster.cc:1428] Setting key f66443f595d77cf9d4c80a44a76e4ab1
I20260928 06:40:26.728102 25351 ts_manager.cc:194] Registered new tserver with Master: becf12ef2f814e66b83a1ddee4fa138b (127.18.211.66:44887)
I20260928 06:40:26.728742 25351 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.211.66@KRBTEST.COM'} at 127.18.211.66:54499
WARNING: no policy specified for oryx/127.18.211.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.211.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.211.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.67.keytab.
Entry for principal oryx/127.18.211.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.67.keytab.
I20260928 06:40:26.750149 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.67:0
--local_ip_for_outbound_sockets=127.18.211.67
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:44085
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.67.keytab
--principal=oryx/127.18.211.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:26.829561 25681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:26.829757 25681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:26.829778 25681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:26.830850 25681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:26.830888 25681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:26.830912 25681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.67
I20260928 06:40:26.832298 25681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.67.keytab
--principal=oryx/127.18.211.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:44085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.67
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:26.832487 25681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:26.832655 25681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:26.837489 25687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:26.837495 25689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:26.837677 25681 server_base.cc:1034] running on GCE node
W20260928 06:40:26.837553 25686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:26.838012 25681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:26.838248 25681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:26.839416 25681 hybrid_clock.cc:648] HybridClock initialized: now 1790577626839410 us; error 39 us; skew 500 ppm
Sep 28 06:40:26 dist-test-slave-b28r krb5kdc[25306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577626, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:26.841814 25681 init.cc:377] Logged in from keytab as oryx/127.18.211.67@KRBTEST.COM (short username oryx)
I20260928 06:40:26.842254 25681 webserver.cc:460] Webserver started at http://127.18.211.67:41715/ using document root <none> and password file <none>
I20260928 06:40:26.842459 25681 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:26.842502 25681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:26.842619 25681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:26.843761 25681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-2/data/instance:
uuid: "a25c0921c520499aa06c828f01791c1d"
format_stamp: "Formatted at 2026-09-28 06:40:26 on dist-test-slave-b28r"
server_key: "410455ad97777eb86fade27764db050e"
server_key_iv: "9b0dd349c27cd84662369fb26560e962"
server_key_version: "encryptionkey@0"
I20260928 06:40:26.844062 25681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance:
uuid: "a25c0921c520499aa06c828f01791c1d"
format_stamp: "Formatted at 2026-09-28 06:40:26 on dist-test-slave-b28r"
server_key: "410455ad97777eb86fade27764db050e"
server_key_iv: "9b0dd349c27cd84662369fb26560e962"
server_key_version: "encryptionkey@0"
I20260928 06:40:26.845348 25681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:40:26.846014 25696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:26.846165 25681 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:40:26.846244 25681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-2/wal
uuid: "a25c0921c520499aa06c828f01791c1d"
format_stamp: "Formatted at 2026-09-28 06:40:26 on dist-test-slave-b28r"
server_key: "410455ad97777eb86fade27764db050e"
server_key_iv: "9b0dd349c27cd84662369fb26560e962"
server_key_version: "encryptionkey@0"
I20260928 06:40:26.846349 25681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:26.885883 25681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:26.887912 25681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:26.888041 25681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:26.888284 25681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:26.888592 25681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:26.888626 25681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:26.888653 25681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:26.888679 25681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:26.894102 25681 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.67:39055
I20260928 06:40:26.894155 25809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.67:39055 every 8 connection(s)
I20260928 06:40:26.894464 25681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
I20260928 06:40:26.895125 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 25681
I20260928 06:40:26.895219 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance
I20260928 06:40:26.895407 19277 external_mini_cluster.cc:1428] Setting key 6b2e7f87bd5d54924587c85d4ef12f24
Sep 28 06:40:26 dist-test-slave-b28r krb5kdc[25306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577626, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM
I20260928 06:40:26.903182 25409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:26.895148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:52403 (local address 127.18.211.126:44085)
0928 06:40:26.895212 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:26.895215 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:26.895428 (+ 213us) server_negotiation.cc:407] Connection header received
0928 06:40:26.896045 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:26.896046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:26.896058 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:26.896115 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:26.897115 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:26.897664 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:26.898615 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:26.898819 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:26.898874 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:26.900878 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:26.900879 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:26.900880 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:26.900886 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:26.901934 (+ 1048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:26.902357 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:26.902358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:26.902359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:26.902412 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:26.902612 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:26.902613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:26.902613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:26.902707 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:26.902752 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:26.903031 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 06:40:26.903110 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 06:40:26.903403 25812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:26.895115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44085 (local address 127.18.211.67:52403)
0928 06:40:26.895276 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:26.895336 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:26.895946 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:26.896191 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:26.896195 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:26.896801 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:26.897002 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:26.897006 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:26.897739 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:26.897740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:26.898525 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:26.898527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:26.898883 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:26.898883 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:26.898982 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:26.899460 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:26.899468 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:26.900783 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:26.902002 (+ 1219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:26.902008 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:26.902009 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:26.902221 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:26.902462 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:26.902462 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:26.902463 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:26.902569 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:26.902781 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:26.902787 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:26.902915 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 06:40:26.903013 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 06:40:26.903171 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":81,"threads_started":1}
I20260928 06:40:26.903885 25810 heartbeater.cc:347] Connected to a master server at 127.18.211.126:44085
I20260928 06:40:26.903972 25810 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:26.904150 25810 heartbeater.cc:511] Master 127.18.211.126:44085 requested a full tablet report, sending...
I20260928 06:40:26.904563 25351 ts_manager.cc:194] Registered new tserver with Master: a25c0921c520499aa06c828f01791c1d (127.18.211.67:39055)
I20260928 06:40:26.904901 25351 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.211.67@KRBTEST.COM'} at 127.18.211.67:52403
I20260928 06:40:26.907246 19277 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.18.211.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.211.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.211.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.125.keytab.
Entry for principal oryx/127.18.211.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.125.keytab.
I20260928 06:40:26.930020 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.125:37143
--webserver_interface=127.18.211.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.125.keytab
--principal=oryx/127.18.211.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:44085,127.18.211.125:37143
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:27.008904 25815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:27.009121 25815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:27.009145 25815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:27.010301 25815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:40:27.010344 25815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:27.010357 25815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:27.010368 25815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:40:27.010380 25815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:40:27.011844 25815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:44085,127.18.211.125:37143
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.125.keytab
--principal=oryx/127.18.211.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.125:37143
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.211.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
W20260928 06:40:27.011948 25815 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.18.211.126:44085,127.18.211.125:37143'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 06:40:27.012100 25815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:27.012308 25815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:27.016840 25823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:27.016853 25820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:27.016848 25821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:27.017202 25815 server_base.cc:1034] running on GCE node
I20260928 06:40:27.017439 25815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:27.017727 25815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:27.018884 25815 hybrid_clock.cc:648] HybridClock initialized: now 1790577627018872 us; error 29 us; skew 500 ppm
Sep 28 06:40:27 dist-test-slave-b28r krb5kdc[25306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577627, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:27.021528 25815 init.cc:377] Logged in from keytab as oryx/127.18.211.125@KRBTEST.COM (short username oryx)
I20260928 06:40:27.022068 25815 webserver.cc:460] Webserver started at http://127.18.211.125:46587/ using document root <none> and password file <none>
I20260928 06:40:27.022322 25815 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:27.022382 25815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:27.022519 25815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:27.023654 25815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-1/data/instance:
uuid: "1295d210e86447f6ba916d9371e50885"
format_stamp: "Formatted at 2026-09-28 06:40:27 on dist-test-slave-b28r"
server_key: "50e2741e76edc7bb9ee06d0d06e28293"
server_key_iv: "152d454c2cedcd7f2b7988e52918bf14"
server_key_version: "encryptionkey@0"
I20260928 06:40:27.024003 25815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-1/wal/instance:
uuid: "1295d210e86447f6ba916d9371e50885"
format_stamp: "Formatted at 2026-09-28 06:40:27 on dist-test-slave-b28r"
server_key: "50e2741e76edc7bb9ee06d0d06e28293"
server_key_iv: "152d454c2cedcd7f2b7988e52918bf14"
server_key_version: "encryptionkey@0"
I20260928 06:40:27.025389 25815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:40:27.026140 25830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:27.026404 25815 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:27.026508 25815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-1/wal
uuid: "1295d210e86447f6ba916d9371e50885"
format_stamp: "Formatted at 2026-09-28 06:40:27 on dist-test-slave-b28r"
server_key: "50e2741e76edc7bb9ee06d0d06e28293"
server_key_iv: "152d454c2cedcd7f2b7988e52918bf14"
server_key_version: "encryptionkey@0"
I20260928 06:40:27.026585 25815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:27.034695 25815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:27.036585 25815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:27.036711 25815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:27.040468 25815 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.125:37143
I20260928 06:40:27.040501 25882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.125:37143 every 8 connection(s)
I20260928 06:40:27.041026 25815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-1/data/info.pb
I20260928 06:40:27.041968 25883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:27.043704 25883 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:40:27.044358 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 25815
Sep 28 06:40:27 dist-test-slave-b28r krb5kdc[25306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577627, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.125@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM
Sep 28 06:40:27 dist-test-slave-b28r krb5kdc[25306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.211.125@KRBTEST.COM
I20260928 06:40:27.053429 25409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.044422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43428 (local address 127.18.211.126:44085)
0928 06:40:27.044503 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:27.044505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:27.044720 (+ 215us) server_negotiation.cc:407] Connection header received
0928 06:40:27.045248 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:27.045250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:27.045263 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:27.045324 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:27.046029 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.046857 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.047827 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.048023 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.048079 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.050393 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:27.050394 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:27.050394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:27.050401 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:27.051838 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.052308 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.052309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.052310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.052363 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.052600 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.052600 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.052601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.052728 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:27.052802 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:27.053306 (+ 504us) server_negotiation.cc:299] Negotiation successful
0928 06:40:27.053368 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 06:40:27.053648 25885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.044412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44085 (local address 127.0.0.1:43428)
0928 06:40:27.044585 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:27.044622 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:27.045148 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:27.045343 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:27.045347 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:27.045713 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:27.045939 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.045946 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.046942 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.046943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.047738 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.047740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.048090 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.048091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.048202 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.048684 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:27.048694 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:27.050311 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:27.051924 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.051931 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.051933 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.052228 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.052421 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.052422 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.052423 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.052541 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.053028 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:27.053032 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:27.053186 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 06:40:27.053283 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 06:40:27.053429 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":88,"threads_started":1}
I20260928 06:40:27.054018 25887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.044820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56684 (local address 127.18.211.125:37143)
0928 06:40:27.045131 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:27.045134 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:27.045159 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:40:27.045282 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:27.045285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:27.045308 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:27.045398 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:27.046100 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.047294 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.048018 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.048320 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.048376 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.050989 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:27.050996 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:27.050996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:27.051002 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:27.052727 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.053023 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.053026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.053026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.053082 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.053272 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.053273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.053273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.053418 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:27.053557 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:27.053844 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 06:40:27.053919 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":92,"threads_started":1}
I20260928 06:40:27.054258 25883 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:40:27 dist-test-slave-b28r krb5kdc[25306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577627, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.125@KRBTEST.COM for oryx/127.18.211.125@KRBTEST.COM
I20260928 06:40:27.059926 25885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.054503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.125:37143 (local address 127.0.0.1:56692)
0928 06:40:27.054546 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:27.054556 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:27.054594 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:27.054716 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:27.054717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:27.054823 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:27.054850 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.054851 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.055482 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.055482 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.055960 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.055961 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.056240 (+ 279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.056241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.056279 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.056564 (+ 285us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:27.056568 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:27.057679 (+ 1111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:27.058997 (+ 1318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.058997 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.058998 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.059195 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.059406 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.059407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.059407 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.059482 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.059670 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:27.059671 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:27.059707 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 06:40:27.059763 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 06:40:27.059843 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 06:40:27.059978 25887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.054547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56692 (local address 127.18.211.125:37143)
0928 06:40:27.054639 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:27.054640 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:27.054647 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:40:27.054663 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:27.054664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:27.054673 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:27.054711 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:27.054898 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.055411 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.056012 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.056196 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.056238 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.057756 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:27.057757 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:27.057757 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:27.057762 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:27.058915 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.059254 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.059255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.059255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.059326 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.059546 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.059546 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.059547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.059626 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:27.059665 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:27.059779 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 06:40:27.059883 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":73}
I20260928 06:40:27.061204 25883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885: Bootstrap starting.
I20260928 06:40:27.062029 25883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:27.062292 25883 log.cc:824] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:27.063010 25883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885: No bootstrap required, opened a new log
I20260928 06:40:27.064410 25883 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } }
I20260928 06:40:27.064528 25883 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:27.064550 25883 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1295d210e86447f6ba916d9371e50885, State: Initialized, Role: FOLLOWER
I20260928 06:40:27.064687 25883 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } }
I20260928 06:40:27.064983 25883 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:40:27.064945 25889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } } }
I20260928 06:40:27.065043 25889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 06:40:27.066473 25900 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:40:27.066522 25900 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:40:27.068179 25815 master_runner.cc:428] Detected that this master 1295d210e86447f6ba916d9371e50885 is joining an existing cluster
I20260928 06:40:27.068221 25815 master_runner.cc:432] Initiating AddMaster RPC to add 127.18.211.125:37143
I20260928 06:40:27.075537 25887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.069360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56702 (local address 127.18.211.125:37143)
0928 06:40:27.069394 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:27.069396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:27.069495 (+ 99us) server_negotiation.cc:407] Connection header received
0928 06:40:27.069562 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:27.069563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:27.069573 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:27.069618 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:27.069972 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.070482 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.071352 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.071656 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.071730 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.072804 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:27.072805 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:27.072805 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:27.072813 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:27.074400 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.074724 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.074725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.074726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.074787 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.074973 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.074974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.074974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.075120 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:27.075181 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:27.075346 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 06:40:27.075422 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:40:27.075560 25409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.069192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43432 (local address 127.18.211.126:44085)
0928 06:40:27.069236 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:27.069238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:27.069364 (+ 126us) server_negotiation.cc:407] Connection header received
0928 06:40:27.069425 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:27.069426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:27.069440 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:27.069503 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:27.069959 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.070807 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.071764 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.072023 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.072055 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.072981 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:27.072982 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:27.072982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:27.072989 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:27.074285 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.074669 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.074670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.074670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.074727 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.074915 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.074916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.074916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.075042 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:27.075074 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:27.075400 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 06:40:27.075480 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:40:27.075847 25906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.069174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.125:37143 (local address 127.0.0.1:56702)
0928 06:40:27.069448 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:27.069456 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:27.069494 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:27.069621 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:27.069622 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:27.069843 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:27.069896 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.069898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.070546 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.070547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.071274 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.071276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.071741 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.071742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.071802 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.072287 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:27.072292 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:27.072721 (+ 429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:27.074469 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.074470 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.074470 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.074660 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.074851 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.074852 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.074852 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.074915 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.075205 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:27.075206 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:27.075241 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 06:40:27.075496 (+ 255us) client_negotiation.cc:240] Negotiation successful
0928 06:40:27.075571 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":75,"threads_started":1}
I20260928 06:40:27.075937 25907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.069148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44085 (local address 127.0.0.1:43432)
0928 06:40:27.069297 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:27.069307 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:27.069368 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:27.069522 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:27.069523 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:27.069776 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:27.069876 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.069878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.070900 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.070901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.071678 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.071680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.072143 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.072144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.072205 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.072520 (+ 315us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:27.072524 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:27.072913 (+ 389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:27.074362 (+ 1449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.074363 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.074364 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.074582 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.074791 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.074792 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.074792 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.074844 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.075142 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:27.075143 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:27.075184 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 06:40:27.075365 (+ 181us) client_negotiation.cc:240] Negotiation successful
0928 06:40:27.075480 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":35,"thread_start_us":74,"threads_started":1}
Sep 28 06:40:27 dist-test-slave-b28r krb5kdc[25306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577626, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.126@KRBTEST.COM for oryx/127.18.211.125@KRBTEST.COM
I20260928 06:40:27.085150 25887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.078129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56718 (local address 127.18.211.125:37143)
0928 06:40:27.078168 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:27.078170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:27.078352 (+ 182us) server_negotiation.cc:407] Connection header received
0928 06:40:27.078413 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:27.078414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:27.078421 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:27.078460 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:27.078951 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.079457 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.080540 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.080716 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.080771 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.082830 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:27.082831 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:27.082832 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:27.082838 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:27.084111 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.084459 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.084460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.084460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.084510 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.084667 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.084668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.084668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.084766 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:27.084827 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:27.084987 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 06:40:27.085058 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:40:27.085160 25908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.078068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.125:37143 (local address 127.0.0.1:56718)
0928 06:40:27.078205 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:27.078265 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:27.078342 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:27.078477 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:27.078480 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:27.078827 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:27.078886 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.078889 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.079522 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.079523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.080458 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.080461 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.080785 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.080786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.080870 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.081260 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:27.081264 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:27.082743 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:27.084185 (+ 1442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.084192 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.084194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.084395 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.084562 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.084563 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.084563 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.084619 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.084826 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:27.084834 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:27.084885 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 06:40:27.084957 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 06:40:27.085069 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
I20260928 06:40:27.085712 25351 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.18.211.125:37143: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1295d210e86447f6ba916d9371e50885"
member_type: NON_VOTER
last_known_addr {
host: "127.18.211.125"
port: 37143
}
attrs {
promote: true
}
}
dest_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be"
cas_config_opid_index: -1
I20260928 06:40:27.085999 25351 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: NON_VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: true } }
I20260928 06:40:27.086452 25910 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1295d210e86447f6ba916d9371e50885 (127.18.211.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: NON_VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: true } } }
I20260928 06:40:27.086798 25857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:27.087553 25910 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.18.211.125:37143
I20260928 06:40:27.087721 25857 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [term 1 FOLLOWER]: Refusing update from remote peer f5d90cb26f1a4900a3e9a927bb0d79be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 06:40:27.087773 25815 master.cc:537] Master@127.18.211.125:37143 shutting down...
I20260928 06:40:27.087862 25909 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: NON_VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: true } } }
I20260928 06:40:27.088042 25909 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:27.088050 25910 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: NON_VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 06:40:27.088773 25337 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 06:40:27.089269 25337 consensus_peers.cc:487] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be -> Peer 1295d210e86447f6ba916d9371e50885 (127.18.211.125:37143): Couldn't send request to peer 1295d210e86447f6ba916d9371e50885. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 06:40:27.090276 25815 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:40:27.090366 25815 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:27.090415 25815 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885: stopping tablet replica
I20260928 06:40:27.102972 25815 master.cc:552] Master@127.18.211.125:37143 shutdown complete.
I20260928 06:40:27.103030 25815 master_runner.cc:305] Clearing existing system tablet
I20260928 06:40:27.103180 25815 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:27.103222 25815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:27.103787 25815 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:40:27.104213 25914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:27.104378 25815 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:27.104460 25815 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:27.104487 25815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-1/wal
uuid: "1295d210e86447f6ba916d9371e50885"
format_stamp: "Formatted at 2026-09-28 06:40:27 on dist-test-slave-b28r"
server_key: "50e2741e76edc7bb9ee06d0d06e28293"
server_key_iv: "152d454c2cedcd7f2b7988e52918bf14"
server_key_version: "encryptionkey@0"
I20260928 06:40:27.105361 25815 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:27.106923 25815 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 06:40:27.106999 25815 log.cc:1196] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 06:40:27.107218 25815 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885: Deleting consensus metadata
I20260928 06:40:27.107345 25815 master_runner.cc:315] Copying system tablet from 127.18.211.126:44085
I20260928 06:40:27.107971 25815 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885: tablet copy: Beginning tablet copy session from remote peer at address 127.18.211.126:44085
I20260928 06:40:27.114316 25920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.108252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44085 (local address 127.0.0.1:43436)
0928 06:40:27.108458 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:27.108467 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:27.108522 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:27.108662 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:27.108662 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:27.108811 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:27.108864 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.108866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.109827 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.109828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.110392 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.110394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.110855 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.110856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.110909 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.111409 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:27.111414 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:27.111700 (+ 286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:27.113393 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.113393 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.113394 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.113557 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.113739 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.113739 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.113739 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.113786 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.114024 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:27.114025 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:27.114061 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 06:40:27.114117 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 06:40:27.114234 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":123,"threads_started":1}
I20260928 06:40:27.114337 25409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.108364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43436 (local address 127.18.211.126:44085)
0928 06:40:27.108427 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:27.108429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:27.108522 (+ 93us) server_negotiation.cc:407] Connection header received
0928 06:40:27.108566 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:27.108567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:27.108578 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:27.108658 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:27.108923 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.109721 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.110470 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.110775 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.110847 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.111788 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:27.111789 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:27.111790 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:27.111798 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:27.113311 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.113620 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.113621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.113621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.113682 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.113834 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.113834 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.113835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.113969 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:27.114029 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:27.114154 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 06:40:27.114233 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 06:40:27.114660 25371 tablet_copy_service.cc:140] P f5d90cb26f1a4900a3e9a927bb0d79be: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1295d210e86447f6ba916d9371e50885 ({username='oryx', principal='oryx/127.18.211.125@KRBTEST.COM'} at 127.0.0.1:43436)
I20260928 06:40:27.114769 25371 tablet_copy_service.cc:161] P f5d90cb26f1a4900a3e9a927bb0d79be: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1295d210e86447f6ba916d9371e50885 at {username='oryx', principal='oryx/127.18.211.125@KRBTEST.COM'} at 127.0.0.1:43436: session id = 1295d210e86447f6ba916d9371e50885-00000000000000000000000000000000
I20260928 06:40:27.115653 25371 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 06:40:27.116310 25815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 06:40:27.117216 25815 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885: tablet copy: Starting download of 0 data blocks...
I20260928 06:40:27.117362 25815 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885: tablet copy: Starting download of 1 WAL segments...
I20260928 06:40:27.118217 25815 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 06:40:27.119043 25371 tablet_copy_service.cc:342] P f5d90cb26f1a4900a3e9a927bb0d79be: Request end of tablet copy session 1295d210e86447f6ba916d9371e50885-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.18.211.125@KRBTEST.COM'} at 127.0.0.1:43436
I20260928 06:40:27.119117 25371 tablet_copy_service.cc:433] P f5d90cb26f1a4900a3e9a927bb0d79be: ending tablet copy session 1295d210e86447f6ba916d9371e50885-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1295d210e86447f6ba916d9371e50885
I20260928 06:40:27.121770 25815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:27.122821 25922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:27.122857 25921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:27.122864 25924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:27.123204 25815 server_base.cc:1034] running on GCE node
I20260928 06:40:27.123307 25815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:27.123441 25815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:27.124563 25815 hybrid_clock.cc:648] HybridClock initialized: now 1790577627124554 us; error 22 us; skew 500 ppm
Sep 28 06:40:27 dist-test-slave-b28r krb5kdc[25306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577627, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:27.125850 25815 init.cc:377] Logged in from keytab as oryx/127.18.211.125@KRBTEST.COM (short username oryx)
I20260928 06:40:27.126120 25815 webserver.cc:460] Webserver started at http://127.18.211.125:45603/ using document root <none> and password file <none>
I20260928 06:40:27.126272 25815 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:27.126303 25815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:27.126852 25815 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:27.127359 25931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:27.127462 25815 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:27.127504 25815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-1/wal
uuid: "1295d210e86447f6ba916d9371e50885"
format_stamp: "Formatted at 2026-09-28 06:40:27 on dist-test-slave-b28r"
server_key: "50e2741e76edc7bb9ee06d0d06e28293"
server_key_iv: "152d454c2cedcd7f2b7988e52918bf14"
server_key_version: "encryptionkey@0"
I20260928 06:40:27.127559 25815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:27.132675 25815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:27.133998 25815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:27.136795 25815 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.125:37143
I20260928 06:40:27.137193 25983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.125:37143 every 8 connection(s)
I20260928 06:40:27.137203 25815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-1/data/info.pb
I20260928 06:40:27.137964 25984 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:40:27.138331 25984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885: Bootstrap starting.
I20260928 06:40:27.140513 25984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:40:27.140713 25984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885: Bootstrap complete.
I20260928 06:40:27.140875 25984 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: NON_VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: true } }
I20260928 06:40:27.140939 25984 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1295d210e86447f6ba916d9371e50885, State: Initialized, Role: LEARNER
I20260928 06:40:27.140964 25984 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: NON_VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: true } }
I20260928 06:40:27.141124 25984 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:40:27.141115 25987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: NON_VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: true } } }
I20260928 06:40:27.141243 25987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [sys.catalog]: This master's current role is: LEARNER
I20260928 06:40:27.142393 25998 catalog_manager.cc:1259] Loaded cluster ID: dce0b6ad27964fd8a9550617d17fad5b
I20260928 06:40:27.142431 25998 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885: loading cluster ID for follower catalog manager: success
I20260928 06:40:27.143388 25998 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885: acquiring CA information for follower catalog manager: success
I20260928 06:40:27.143642 25998 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:40:27.161526 25999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.155448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56736 (local address 127.18.211.125:37143)
0928 06:40:27.155590 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:27.155592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:27.155599 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:40:27.155622 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:27.155622 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:27.155632 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:27.155695 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:27.156247 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.156927 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.157625 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.157825 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.157887 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.159124 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:27.159125 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:27.159126 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:27.159133 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:27.160494 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.160801 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.160802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.160803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.160859 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.161027 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.161028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.161029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.161134 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:27.161218 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:27.161320 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 06:40:27.161401 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
WARNING: no policy specified for oryx/127.18.211.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.211.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.211.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.124.keytab.
Entry for principal oryx/127.18.211.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.124.keytab.
I20260928 06:40:27.183590 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.124:40615
--webserver_interface=127.18.211.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.124.keytab
--principal=oryx/127.18.211.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:44085,127.18.211.125:37143,127.18.211.124:40615
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:27.264797 26002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:27.264995 26002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:27.265015 26002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:27.266077 26002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:40:27.266110 26002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:27.266124 26002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:27.266134 26002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:40:27.266144 26002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:40:27.267612 26002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:44085,127.18.211.125:37143,127.18.211.124:40615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.124.keytab
--principal=oryx/127.18.211.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.124:40615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.211.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:27.267802 26002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:27.267992 26002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:27.272686 26008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:27.272686 26010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:27.272716 26007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:27.272944 26002 server_base.cc:1034] running on GCE node
I20260928 06:40:27.273170 26002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:27.273480 26002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:27.274621 26002 hybrid_clock.cc:648] HybridClock initialized: now 1790577627274609 us; error 27 us; skew 500 ppm
Sep 28 06:40:27 dist-test-slave-b28r krb5kdc[25306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577627, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:27.277139 26002 init.cc:377] Logged in from keytab as oryx/127.18.211.124@KRBTEST.COM (short username oryx)
I20260928 06:40:27.277570 26002 webserver.cc:460] Webserver started at http://127.18.211.124:39743/ using document root <none> and password file <none>
I20260928 06:40:27.277760 26002 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:27.277807 26002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:27.277926 26002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:27.278915 26002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-2/data/instance:
uuid: "90b91ba24e8c4c20853a01477d44c493"
format_stamp: "Formatted at 2026-09-28 06:40:27 on dist-test-slave-b28r"
server_key: "8f8eb40fd795c3e9ec483e1da4edc051"
server_key_iv: "4963c45f4f49de9415061be676847e3d"
server_key_version: "encryptionkey@0"
I20260928 06:40:27.279264 26002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-2/wal/instance:
uuid: "90b91ba24e8c4c20853a01477d44c493"
format_stamp: "Formatted at 2026-09-28 06:40:27 on dist-test-slave-b28r"
server_key: "8f8eb40fd795c3e9ec483e1da4edc051"
server_key_iv: "4963c45f4f49de9415061be676847e3d"
server_key_version: "encryptionkey@0"
I20260928 06:40:27.280529 26002 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:40:27.281268 26017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:27.281440 26002 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:40:27.281525 26002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-2/wal
uuid: "90b91ba24e8c4c20853a01477d44c493"
format_stamp: "Formatted at 2026-09-28 06:40:27 on dist-test-slave-b28r"
server_key: "8f8eb40fd795c3e9ec483e1da4edc051"
server_key_iv: "4963c45f4f49de9415061be676847e3d"
server_key_version: "encryptionkey@0"
I20260928 06:40:27.281587 26002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:27.292742 26002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:27.294853 26002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:27.294996 26002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:27.298669 26002 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.124:40615
I20260928 06:40:27.298710 26069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.124:40615 every 8 connection(s)
I20260928 06:40:27.299122 26002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-2/data/info.pb
I20260928 06:40:27.300105 26070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:27.301599 26070 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:40:27 dist-test-slave-b28r krb5kdc[25306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577627, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.124@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM
I20260928 06:40:27.308233 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 26002
I20260928 06:40:27.310128 25409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.302414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43438 (local address 127.18.211.126:44085)
0928 06:40:27.302476 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:27.302479 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:27.302605 (+ 126us) server_negotiation.cc:407] Connection header received
0928 06:40:27.303113 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:27.303114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:27.303126 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:27.303194 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:27.303832 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.304372 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.305318 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.305513 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.305567 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.307528 (+ 1961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:27.307530 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:27.307530 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:27.307537 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:27.308855 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.309259 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.309260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.309261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.309329 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.309553 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.309554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.309555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.309676 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:27.309726 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:27.310007 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 06:40:27.310066 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 06:40:27.310401 26072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.302262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44085 (local address 127.0.0.1:43438)
0928 06:40:27.302493 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:27.302529 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:27.303013 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:27.303249 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:27.303254 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:27.303561 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:27.303759 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.303764 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.304456 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.304457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.305245 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.305247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.305591 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.305591 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.305685 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.306114 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:27.306122 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:27.307446 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:27.308939 (+ 1493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.308945 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.308947 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.309187 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.309406 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.309407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.309407 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.309499 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.309729 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:27.309733 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:27.309880 (+ 147us) client_negotiation.cc:769] Sending connection context
0928 06:40:27.309989 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 06:40:27.310137 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":127,"threads_started":1}
I20260928 06:40:27.310976 26070 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:40:27 dist-test-slave-b28r krb5kdc[25306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.211.124@KRBTEST.COM
Sep 28 06:40:27 dist-test-slave-b28r krb5kdc[25306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577627, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.124@KRBTEST.COM for oryx/127.18.211.125@KRBTEST.COM
I20260928 06:40:27.317325 26073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.308726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58590 (local address 127.18.211.124:40615)
0928 06:40:27.308875 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:27.308879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:27.308895 (+ 16us) server_negotiation.cc:407] Connection header received
0928 06:40:27.308911 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:27.308913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:27.308925 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:27.308999 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:27.309587 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.310606 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.311542 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.311888 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.311949 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.314430 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:27.314436 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:27.314437 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:27.314442 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:27.316090 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.316458 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.316461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.316462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.316515 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.316691 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.316692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.316692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.316850 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:27.316984 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:27.317111 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 06:40:27.317179 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":83,"threads_started":1}
I20260928 06:40:27.317373 25999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.311301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56746 (local address 127.18.211.125:37143)
0928 06:40:27.311341 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:27.311343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:27.311385 (+ 42us) server_negotiation.cc:407] Connection header received
0928 06:40:27.311453 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:27.311453 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:27.311466 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:27.311509 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:27.311756 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.312245 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.313216 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.313433 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.313498 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.315049 (+ 1551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:27.315050 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:27.315051 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:27.315059 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:27.316337 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.316646 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.316646 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.316647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.316698 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.316869 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.316870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.316871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.316992 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:27.317051 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:27.317216 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 06:40:27.317291 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:40:27.317386 26072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.311250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.125:37143 (local address 127.0.0.1:56746)
0928 06:40:27.311342 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:27.311352 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:27.311398 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:27.311510 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:27.311510 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:27.311631 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:27.311675 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.311676 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.312345 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.312346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.313136 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.313139 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.313504 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.313505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.313564 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.313884 (+ 320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:27.313888 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:27.314970 (+ 1082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:27.316407 (+ 1437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.316408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.316408 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.316583 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.316761 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.316761 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.316762 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.316816 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.317073 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:27.317074 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:27.317113 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 06:40:27.317182 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 06:40:27.317266 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 06:40:27.317945 26070 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.211.124" port: 40615 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:40:27 dist-test-slave-b28r krb5kdc[25306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577627, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.124@KRBTEST.COM for oryx/127.18.211.124@KRBTEST.COM
I20260928 06:40:27.323438 26072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.318158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:40615 (local address 127.0.0.1:58606)
0928 06:40:27.318200 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:27.318210 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:27.318245 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:27.318357 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:27.318358 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:27.318472 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:27.318494 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.318495 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.319050 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.319050 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.319540 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.319541 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.319856 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.319857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.319901 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.320196 (+ 295us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:27.320199 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:27.321241 (+ 1042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:27.322518 (+ 1277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.322519 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.322519 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.322670 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.322866 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.322867 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.322867 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.322920 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.323131 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:27.323132 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:27.323167 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 06:40:27.323228 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 06:40:27.323323 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 06:40:27.323479 26073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.318243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58606 (local address 127.18.211.124:40615)
0928 06:40:27.318284 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:27.318286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:27.318292 (+ 6us) server_negotiation.cc:407] Connection header received
0928 06:40:27.318306 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:27.318306 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:27.318315 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:27.318356 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:27.318549 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.318988 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.319603 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.319803 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.319854 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.321348 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:27.321349 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:27.321349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:27.321356 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:27.322447 (+ 1091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.322730 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.322731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.322731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.322811 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.322968 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.322968 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.322968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.323061 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:27.323139 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:27.323255 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 06:40:27.323371 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":27}
I20260928 06:40:27.324702 26070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493: Bootstrap starting.
I20260928 06:40:27.325546 26070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:27.325809 26070 log.cc:824] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:27.326527 26070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493: No bootstrap required, opened a new log
I20260928 06:40:27.328038 26070 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } } peers { permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 40615 } }
I20260928 06:40:27.328162 26070 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:27.328182 26070 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90b91ba24e8c4c20853a01477d44c493, State: Initialized, Role: FOLLOWER
I20260928 06:40:27.328311 26070 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } } peers { permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 40615 } }
I20260928 06:40:27.328552 26075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } } peers { permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 40615 } } }
I20260928 06:40:27.328608 26070 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:40:27.328651 26075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 06:40:27.330220 26086 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:40:27.330282 26086 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:40:27.332471 26002 master_runner.cc:428] Detected that this master 90b91ba24e8c4c20853a01477d44c493 is joining an existing cluster
I20260928 06:40:27.332511 26002 master_runner.cc:432] Initiating AddMaster RPC to add 127.18.211.124:40615
I20260928 06:40:27.339866 26094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.333664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:40615 (local address 127.0.0.1:58612)
0928 06:40:27.333863 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:27.333871 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:27.333925 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:27.334052 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:27.334053 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:27.334320 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:27.334369 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.334371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.335235 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.335236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.336119 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.336121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.336584 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.336585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.336631 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.337043 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:27.337047 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:27.337374 (+ 327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:27.338909 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.338910 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.338911 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.339068 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.339295 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.339295 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.339296 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.339390 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.339626 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:27.339626 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:27.339672 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 06:40:27.339716 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 06:40:27.339791 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":62,"threads_started":1}
I20260928 06:40:27.339960 26073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.333787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58612 (local address 127.18.211.124:40615)
0928 06:40:27.333819 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:27.333821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:27.333909 (+ 88us) server_negotiation.cc:407] Connection header received
0928 06:40:27.333977 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:27.333978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:27.333985 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:27.334008 (+ 23us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:27.334445 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.335159 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.336208 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.336473 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.336508 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.337449 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:27.337450 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:27.337451 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:27.337457 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:27.338828 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.339167 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.339168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.339169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.339231 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.339445 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.339446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.339446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.339571 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:27.339636 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:27.339741 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 06:40:27.339853 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":19}
I20260928 06:40:27.363379 25409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.333501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43454 (local address 127.18.211.126:44085)
0928 06:40:27.333547 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:27.333549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:27.333666 (+ 117us) server_negotiation.cc:407] Connection header received
0928 06:40:27.333764 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:27.333765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:27.333777 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:27.333835 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:27.334310 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.335106 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.336366 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.336657 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.336699 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.360437 (+ 23738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:27.360439 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:27.360439 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:27.360448 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:27.362157 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.362520 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.362522 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.362522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.362583 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.362775 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.362776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.362776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.362943 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:27.363018 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:27.363194 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 06:40:27.363284 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:40:27.363420 26093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.333469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44085 (local address 127.0.0.1:43454)
0928 06:40:27.333607 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:27.333617 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:27.333692 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:27.333885 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:27.333887 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:27.334124 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:27.334215 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.334217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.335196 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.335197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.336264 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.336267 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.336789 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.336790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.336852 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.359729 (+ 22877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:27.359735 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:27.360272 (+ 537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:27.362230 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.362231 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.362231 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.362428 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.362650 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.362651 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.362651 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.362706 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.363015 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:27.363015 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:27.363067 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 06:40:27.363167 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 06:40:27.363300 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":70,"thread_start_us":71,"threads_started":1}
I20260928 06:40:27.365763 25999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.333652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56762 (local address 127.18.211.125:37143)
0928 06:40:27.333691 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:27.333693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:27.333848 (+ 155us) server_negotiation.cc:407] Connection header received
0928 06:40:27.333942 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:27.333943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:27.333954 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:27.333984 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:27.334455 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.335198 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.336281 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.336561 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.336599 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.360374 (+ 23775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:27.360377 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:27.360377 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:27.360386 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:27.362031 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.364980 (+ 2949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.364981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.364982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.365054 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.365242 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.365243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.365243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.365368 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:27.365426 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:27.365577 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 06:40:27.365676 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:40:27.366181 26092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.333480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.125:37143 (local address 127.0.0.1:56762)
0928 06:40:27.333781 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:27.333789 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:27.333874 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:27.334045 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:27.334046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:27.334307 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:27.334369 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.334371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.335274 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.335275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.336169 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.336172 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.336664 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.336664 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.336717 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.359682 (+ 22965us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:27.359690 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:27.360201 (+ 511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:27.362146 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.362147 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.362148 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.362396 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.365132 (+ 2736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.365133 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.365133 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.365191 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.365429 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:27.365430 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:27.365488 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 06:40:27.365542 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 06:40:27.366073 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":76,"thread_start_us":76,"threads_started":1}
I20260928 06:40:27.368690 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 25413
Sep 28 06:40:27 dist-test-slave-b28r krb5kdc[25306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577626, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.126@KRBTEST.COM for oryx/127.18.211.124@KRBTEST.COM
I20260928 06:40:27.373528 25908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.366879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:40615 (local address 127.0.0.1:58628)
0928 06:40:27.366983 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:27.366993 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:27.367040 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:27.367236 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:27.367237 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:27.367378 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:27.367417 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.367419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.368020 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.368021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.368749 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.368752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.369551 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.369552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.369619 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.369963 (+ 344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:27.369967 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:27.371265 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:27.372637 (+ 1372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.372638 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.372638 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.372797 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.372959 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.372960 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.372960 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.373017 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.373221 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:27.373222 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:27.373267 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 06:40:27.373333 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 06:40:27.373434 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260928 06:40:27.373523 26073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.366935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58628 (local address 127.18.211.124:40615)
0928 06:40:27.366976 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:27.366978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:27.367026 (+ 48us) server_negotiation.cc:407] Connection header received
0928 06:40:27.367124 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:27.367125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:27.367135 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:27.367183 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:27.367486 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.367956 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.369219 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.369493 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.369552 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.371342 (+ 1790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:27.371343 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:27.371343 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:27.371350 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:27.372563 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.372856 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.372857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.372858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.372908 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.373065 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.373066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.373066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.373169 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:27.373222 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:27.373363 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 06:40:27.373432 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:40:27.374264 25351 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.18.211.124:40615: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "90b91ba24e8c4c20853a01477d44c493"
member_type: NON_VOTER
last_known_addr {
host: "127.18.211.124"
port: 40615
}
attrs {
promote: true
}
}
dest_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be"
cas_config_opid_index: 5
I20260928 06:40:27.374501 25351 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: NON_VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: true } } peers { permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: NON_VOTER last_known_addr { host: "127.18.211.124" port: 40615 } attrs { promote: true } }
I20260928 06:40:27.374960 25910 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 90b91ba24e8c4c20853a01477d44c493 (127.18.211.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: NON_VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: true } } peers { permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: NON_VOTER last_known_addr { host: "127.18.211.124" port: 40615 } attrs { promote: true } } }
I20260928 06:40:27.375770 25910 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.18.211.124:40615
I20260928 06:40:27.375835 25910 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: NON_VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: true } } peers { permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: NON_VOTER last_known_addr { host: "127.18.211.124" port: 40615 } attrs { promote: true } } }
I20260928 06:40:27.375906 25910 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:27.376211 26002 master.cc:537] Master@127.18.211.124:40615 shutting down...
I20260928 06:40:27.376245 26044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:27.377470 26044 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [term 1 FOLLOWER]: Refusing update from remote peer f5d90cb26f1a4900a3e9a927bb0d79be: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 06:40:27.378177 25909 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [LEADER]: Connected to new peer: Peer: permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: NON_VOTER last_known_addr { host: "127.18.211.124" port: 40615 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 06:40:27.379012 25335 consensus_peers.cc:487] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be -> Peer 90b91ba24e8c4c20853a01477d44c493 (127.18.211.124:40615): Couldn't send request to peer 90b91ba24e8c4c20853a01477d44c493. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 06:40:27.380811 26002 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:40:27.380900 26002 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:27.380942 26002 tablet_replica.cc:330] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493: stopping tablet replica
I20260928 06:40:27.380961 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.65:35203
--local_ip_for_outbound_sockets=127.18.211.65
--tserver_master_addrs=127.18.211.126:44085
--webserver_port=44001
--webserver_interface=127.18.211.65
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.65.keytab
--principal=oryx/127.18.211.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.18.211.126:44085,127.18.211.125:37143
--tserver_master_addrs=127.18.211.126:44085,127.18.211.125:37143,127.18.211.124:40615 with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:40:27.383766 25908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.379123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.125:37143 (local address 127.0.0.1:56766)
0928 06:40:27.379344 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:27.379354 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:27.379394 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:27.379707 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:27.379708 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:27.379710 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:27.379766 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.379767 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.380643 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.380644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.382052 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.382055 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.383534 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.383535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.383600 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.383625 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 06:40:27.383674 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 06:40:27.383713 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205}
I20260928 06:40:27.384012 25999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.375179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56766 (local address 127.18.211.125:37143)
0928 06:40:27.379194 (+ 4015us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:27.379196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:27.379438 (+ 242us) server_negotiation.cc:407] Connection header received
0928 06:40:27.379463 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:27.379464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:27.379488 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:27.379537 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:27.379839 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.380565 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.382132 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.383364 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.383427 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.383499 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:27.383905 (+ 406us) server_negotiation.cc:299] Negotiation successful
0928 06:40:27.383949 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3984}
I20260928 06:40:27.384207 25958 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [term 1 LEARNER]: Refusing update from remote peer f5d90cb26f1a4900a3e9a927bb0d79be: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 06:40:27.384564 25909 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: NON_VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 06:40:27.385093 25958 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 90b91ba24e8c4c20853a01477d44c493 (127.18.211.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: NON_VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: true } } peers { permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: NON_VOTER last_known_addr { host: "127.18.211.124" port: 40615 } attrs { promote: true } } }
I20260928 06:40:27.386173 25909 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be: attempting to promote NON_VOTER 1295d210e86447f6ba916d9371e50885 to VOTER
I20260928 06:40:27.386226 25987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5d90cb26f1a4900a3e9a927bb0d79be. Latest consensus state: current_term: 1 leader_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: NON_VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: true } } peers { permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: NON_VOTER last_known_addr { host: "127.18.211.124" port: 40615 } attrs { promote: true } } }
I20260928 06:40:27.386314 25987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [sys.catalog]: This master's current role is: LEARNER
I20260928 06:40:27.386348 25987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: NON_VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: true } } peers { permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: NON_VOTER last_known_addr { host: "127.18.211.124" port: 40615 } attrs { promote: true } } }
I20260928 06:40:27.386381 25987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [sys.catalog]: This master's current role is: LEARNER
I20260928 06:40:27.386376 25909 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: false } } peers { permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: NON_VOTER last_known_addr { host: "127.18.211.124" port: 40615 } attrs { promote: true } }
I20260928 06:40:27.386876 25908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.386753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:40615 (local address 127.0.0.1:58640)
0928 06:40:27.386802 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:27.386847 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.124:40615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 06:40:27.386917 25958 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [term 1 LEARNER]: Refusing update from remote peer f5d90cb26f1a4900a3e9a927bb0d79be: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 06:40:27.387140 26095 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 06:40:27.387326 25908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.387237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:40615 (local address 127.0.0.1:58642)
0928 06:40:27.387271 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:27.387304 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.124:40615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260928 06:40:27.387382 25335 consensus_peers.cc:487] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be -> Peer 90b91ba24e8c4c20853a01477d44c493 (127.18.211.124:40615): Couldn't send request to peer 90b91ba24e8c4c20853a01477d44c493. Status: Network error: Client connection negotiation failed: client connection to 127.18.211.124:40615: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 06:40:27.387818 26095 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1295d210e86447f6ba916d9371e50885 (127.18.211.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: false } } peers { permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: NON_VOTER last_known_addr { host: "127.18.211.124" port: 40615 } attrs { promote: true } } }
I20260928 06:40:27.387974 25958 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1295d210e86447f6ba916d9371e50885 (127.18.211.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: false } } peers { permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: NON_VOTER last_known_addr { host: "127.18.211.124" port: 40615 } attrs { promote: true } } }
I20260928 06:40:27.388705 26095 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: false } } peers { permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: NON_VOTER last_known_addr { host: "127.18.211.124" port: 40615 } attrs { promote: true } } }
I20260928 06:40:27.388780 26095 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:27.388825 26095 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: false } } peers { permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: NON_VOTER last_known_addr { host: "127.18.211.124" port: 40615 } attrs { promote: true } } }
I20260928 06:40:27.388859 26095 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:27.389094 26098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: false } } peers { permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: NON_VOTER last_known_addr { host: "127.18.211.124" port: 40615 } attrs { promote: true } } }
I20260928 06:40:27.389158 26098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:40:27.394479 26002 master.cc:552] Master@127.18.211.124:40615 shutdown complete.
I20260928 06:40:27.394519 26002 master_runner.cc:305] Clearing existing system tablet
I20260928 06:40:27.394639 26002 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:27.394676 26002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:27.395330 26002 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:40:27.395751 26100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:27.395879 26002 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:27.395928 26002 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:27.395952 26002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-2/wal
uuid: "90b91ba24e8c4c20853a01477d44c493"
format_stamp: "Formatted at 2026-09-28 06:40:27 on dist-test-slave-b28r"
server_key: "8f8eb40fd795c3e9ec483e1da4edc051"
server_key_iv: "4963c45f4f49de9415061be676847e3d"
server_key_version: "encryptionkey@0"
I20260928 06:40:27.396514 26002 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:27.397660 26002 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 06:40:27.397716 26002 log.cc:1196] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 06:40:27.397886 26002 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493: Deleting consensus metadata
I20260928 06:40:27.397993 26002 master_runner.cc:315] Copying system tablet from 127.18.211.126:44085
I20260928 06:40:27.398528 26002 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493: tablet copy: Beginning tablet copy session from remote peer at address 127.18.211.126:44085
I20260928 06:40:27.404138 26106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.398799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44085 (local address 127.0.0.1:43470)
0928 06:40:27.398903 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:27.398914 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:27.398971 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:27.399185 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:27.399186 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:27.399351 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:27.399394 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.399396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.400028 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.400029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.400550 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.400551 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.400887 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.400888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.400940 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.401360 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:27.401365 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:27.401680 (+ 315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:27.403158 (+ 1478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.403160 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.403161 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.403334 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.403503 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.403504 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.403504 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.403561 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.403841 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:27.403841 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:27.403880 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 06:40:27.403931 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 06:40:27.404040 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":66,"threads_started":1}
I20260928 06:40:27.404135 25409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.398919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43470 (local address 127.18.211.126:44085)
0928 06:40:27.398965 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:27.398967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:27.398975 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:40:27.399032 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:27.399033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:27.399043 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:27.399179 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:27.399467 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.399959 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.400620 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.400825 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.400884 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.401766 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:27.401768 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:27.401768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:27.401776 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:27.403045 (+ 1269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.403395 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.403396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.403397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.403450 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.403631 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.403632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.403632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.403760 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:27.403804 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:27.403971 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 06:40:27.404047 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:40:27.404593 25371 tablet_copy_service.cc:140] P f5d90cb26f1a4900a3e9a927bb0d79be: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 90b91ba24e8c4c20853a01477d44c493 ({username='oryx', principal='oryx/127.18.211.124@KRBTEST.COM'} at 127.0.0.1:43470)
I20260928 06:40:27.404671 25371 tablet_copy_service.cc:161] P f5d90cb26f1a4900a3e9a927bb0d79be: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 90b91ba24e8c4c20853a01477d44c493 at {username='oryx', principal='oryx/127.18.211.124@KRBTEST.COM'} at 127.0.0.1:43470: session id = 90b91ba24e8c4c20853a01477d44c493-00000000000000000000000000000000
I20260928 06:40:27.405239 25371 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 06:40:27.405678 26002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 06:40:27.406584 26002 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493: tablet copy: Starting download of 0 data blocks...
I20260928 06:40:27.406713 26002 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493: tablet copy: Starting download of 1 WAL segments...
I20260928 06:40:27.407409 26002 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 06:40:27.408131 25371 tablet_copy_service.cc:342] P f5d90cb26f1a4900a3e9a927bb0d79be: Request end of tablet copy session 90b91ba24e8c4c20853a01477d44c493-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.18.211.124@KRBTEST.COM'} at 127.0.0.1:43470
I20260928 06:40:27.408183 25371 tablet_copy_service.cc:433] P f5d90cb26f1a4900a3e9a927bb0d79be: ending tablet copy session 90b91ba24e8c4c20853a01477d44c493-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 90b91ba24e8c4c20853a01477d44c493
I20260928 06:40:27.410176 26002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:27.411207 26107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:27.411501 26002 server_base.cc:1034] running on GCE node
W20260928 06:40:27.411652 26110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:27.415558 26108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:27.415686 26002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:27.415815 26002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:27.416932 26002 hybrid_clock.cc:648] HybridClock initialized: now 1790577627416922 us; error 31 us; skew 500 ppm
Sep 28 06:40:27 dist-test-slave-b28r krb5kdc[25306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577627, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:27.418120 26002 init.cc:377] Logged in from keytab as oryx/127.18.211.124@KRBTEST.COM (short username oryx)
I20260928 06:40:27.418363 26002 webserver.cc:460] Webserver started at http://127.18.211.124:33469/ using document root <none> and password file <none>
I20260928 06:40:27.418476 26002 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:27.418509 26002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:27.419109 26002 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:27.419592 26117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:27.419689 26002 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:27.419727 26002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-2/wal
uuid: "90b91ba24e8c4c20853a01477d44c493"
format_stamp: "Formatted at 2026-09-28 06:40:27 on dist-test-slave-b28r"
server_key: "8f8eb40fd795c3e9ec483e1da4edc051"
server_key_iv: "4963c45f4f49de9415061be676847e3d"
server_key_version: "encryptionkey@0"
I20260928 06:40:27.419783 26002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:27.425701 26002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:27.427224 26002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:27.430013 26002 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.124:40615
I20260928 06:40:27.430368 26002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/master-2/data/info.pb
I20260928 06:40:27.431241 26169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.124:40615 every 8 connection(s)
I20260928 06:40:27.431864 26170 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:40:27.432276 26170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493: Bootstrap starting.
I20260928 06:40:27.434566 26170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:40:27.434773 26170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493: Bootstrap complete.
I20260928 06:40:27.434949 26170 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: false } } peers { permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: NON_VOTER last_known_addr { host: "127.18.211.124" port: 40615 } attrs { promote: true } }
I20260928 06:40:27.435021 26170 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 90b91ba24e8c4c20853a01477d44c493, State: Initialized, Role: LEARNER
I20260928 06:40:27.435056 26170 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: false } } peers { permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: NON_VOTER last_known_addr { host: "127.18.211.124" port: 40615 } attrs { promote: true } }
I20260928 06:40:27.435249 26170 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:40:27.435243 26173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: false } } peers { permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: NON_VOTER last_known_addr { host: "127.18.211.124" port: 40615 } attrs { promote: true } } }
I20260928 06:40:27.435485 26173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [sys.catalog]: This master's current role is: LEARNER
I20260928 06:40:27.436614 26184 catalog_manager.cc:1259] Loaded cluster ID: dce0b6ad27964fd8a9550617d17fad5b
I20260928 06:40:27.437359 26184 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493: loading cluster ID for follower catalog manager: success
I20260928 06:40:27.438097 26184 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493: acquiring CA information for follower catalog manager: success
I20260928 06:40:27.438341 26184 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 06:40:27.495276 26096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:27.495483 26096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:27.495508 26096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:27.496693 26096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:27.496737 26096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:27.496768 26096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.65
I20260928 06:40:27.498275 26096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.65.keytab
--principal=oryx/127.18.211.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.65:35203
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.211.65
--webserver_port=44001
--tserver_master_addrs=127.18.211.126:44085,127.18.211.125:37143,127.18.211.124:40615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.65
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:27.498492 26096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:27.498723 26096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:27.503448 26190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:27.503522 26189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:27.503574 26192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:27.504007 26096 server_base.cc:1034] running on GCE node
I20260928 06:40:27.504208 26096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:27.504415 26096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:27.505559 26096 hybrid_clock.cc:648] HybridClock initialized: now 1790577627505546 us; error 29 us; skew 500 ppm
Sep 28 06:40:27 dist-test-slave-b28r krb5kdc[25306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577627, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:27.507952 26096 init.cc:377] Logged in from keytab as oryx/127.18.211.65@KRBTEST.COM (short username oryx)
I20260928 06:40:27.508347 26096 webserver.cc:460] Webserver started at http://127.18.211.65:44001/ using document root <none> and password file <none>
I20260928 06:40:27.508530 26096 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:27.508587 26096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:27.509984 26096 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:40:27.510556 26199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:27.510710 26096 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:40:27.510804 26096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-0/wal
uuid: "538426449b59422daa1419fc97acf5a6"
format_stamp: "Formatted at 2026-09-28 06:40:26 on dist-test-slave-b28r"
server_key: "d689cfc63c3d15c336a039dac7b5c945"
server_key_iv: "c7fd1c9ba107bb235a3d0df2e091ca45"
server_key_version: "encryptionkey@0"
I20260928 06:40:27.511173 26096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:27.525646 26096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:27.527926 26096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:27.528069 26096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:27.528306 26096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:27.528632 26096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:27.528667 26096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:27.528697 26096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:27.528721 26096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:27.534749 26096 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.65:35203
I20260928 06:40:27.534809 26312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.65:35203 every 8 connection(s)
I20260928 06:40:27.535197 26096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
I20260928 06:40:27.537135 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 26096
I20260928 06:40:27.537307 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 25547
Sep 28 06:40:27 dist-test-slave-b28r krb5kdc[25306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577627, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.124@KRBTEST.COM
I20260928 06:40:27.543128 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.66:44887
--local_ip_for_outbound_sockets=127.18.211.66
--tserver_master_addrs=127.18.211.126:44085
--webserver_port=34153
--webserver_interface=127.18.211.66
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.66.keytab
--principal=oryx/127.18.211.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.18.211.126:44085,127.18.211.125:37143
--tserver_master_addrs=127.18.211.126:44085,127.18.211.125:37143,127.18.211.124:40615 with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:40:27 dist-test-slave-b28r krb5kdc[25306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577627, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM
I20260928 06:40:27.546802 25409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.536122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:34931 (local address 127.18.211.126:44085)
0928 06:40:27.536187 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:27.536188 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:27.536541 (+ 353us) server_negotiation.cc:407] Connection header received
0928 06:40:27.538359 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:27.538361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:27.538375 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:27.538433 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:27.538763 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.539492 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.540670 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.540952 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.541015 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.543883 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:27.543884 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:27.543884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:27.543892 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:27.545423 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.545872 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.545873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.545874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.545931 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.546102 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.546103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.546103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.546219 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:27.546258 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:27.546620 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 06:40:27.546704 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:40:27.546963 26319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.536372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:53869 (local address 127.18.211.124:40615)
0928 06:40:27.536498 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:27.536500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:27.536611 (+ 111us) server_negotiation.cc:407] Connection header received
0928 06:40:27.536948 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:27.536949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:27.536961 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:27.537005 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:27.538416 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.539442 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.540748 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.541046 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.541098 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.543568 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:27.543570 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:27.543571 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:27.543578 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:27.545293 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.545781 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.545782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.545783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.545841 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.546083 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.546084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.546084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.546212 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:27.546286 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:27.546799 (+ 513us) server_negotiation.cc:299] Negotiation successful
0928 06:40:27.546879 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":55,"threads_started":1}
I20260928 06:40:27.547013 26320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.536284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:40615 (local address 127.18.211.65:53869)
0928 06:40:27.536558 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:27.536567 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:27.536843 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:27.537123 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:27.537126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:27.538123 (+ 997us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:27.538320 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.538326 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.539638 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.539639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.540677 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.540680 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.541111 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.541112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.541176 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.541850 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:27.541854 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:27.543459 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:27.545393 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.545399 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.545402 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.545712 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.545897 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.545898 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.545898 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.546018 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.546314 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:27.546315 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:27.546459 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 06:40:27.546560 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 06:40:27.546776 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":31,"thread_start_us":199,"threads_started":1}
I20260928 06:40:27.547508 26313 heartbeater.cc:347] Connected to a master server at 127.18.211.124:40615
I20260928 06:40:27.547612 26313 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:27.547734 26317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.535997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44085 (local address 127.18.211.65:34931)
0928 06:40:27.536469 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:27.536481 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:27.536840 (+ 359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:27.538487 (+ 1647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:27.538488 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:27.538630 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:27.538689 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.538691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.539577 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.539578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.540594 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.540597 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.541029 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.541029 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.541161 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.541761 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:27.541770 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:27.543782 (+ 2012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:27.545525 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.545526 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.545526 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.545792 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.545993 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.545993 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.545994 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.546045 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.546297 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:27.546304 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:27.546458 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 06:40:27.547461 (+ 1003us) client_negotiation.cc:240] Negotiation successful
0928 06:40:27.547556 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":56,"thread_start_us":83,"threads_started":1}
I20260928 06:40:27.547811 26313 heartbeater.cc:511] Master 127.18.211.124:40615 requested a full tablet report, sending...
Sep 28 06:40:27 dist-test-slave-b28r krb5kdc[25306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577627, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.125@KRBTEST.COM
I20260928 06:40:27.548324 26133 ts_manager.cc:194] Registered new tserver with Master: 538426449b59422daa1419fc97acf5a6 (127.18.211.65:35203)
I20260928 06:40:27.548398 26314 heartbeater.cc:347] Connected to a master server at 127.18.211.126:44085
I20260928 06:40:27.548440 26314 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:27.548501 26314 heartbeater.cc:511] Master 127.18.211.126:44085 requested a full tablet report, sending...
I20260928 06:40:27.549018 25351 ts_manager.cc:194] Re-registered known tserver with Master: 538426449b59422daa1419fc97acf5a6 (127.18.211.65:35203)
I20260928 06:40:27.549476 25351 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.211.65@KRBTEST.COM'} at 127.18.211.65:34931
I20260928 06:40:27.551074 25999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.535988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:42887 (local address 127.18.211.125:37143)
0928 06:40:27.536043 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:27.536044 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:27.536304 (+ 260us) server_negotiation.cc:407] Connection header received
0928 06:40:27.536926 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:27.536927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:27.536940 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:27.537006 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:27.543104 (+ 6098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.544082 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.545699 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.545997 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.546039 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.548460 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:27.548461 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:27.548461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:27.548469 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:27.550041 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.550391 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.550392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.550392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.550449 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.550607 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.550608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.550608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.550729 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:27.550785 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:27.550915 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 06:40:27.550989 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:40:27.551138 26318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.536003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.125:37143 (local address 127.18.211.65:42887)
0928 06:40:27.536171 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:27.536214 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:27.536831 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:27.537124 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:27.537127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:27.542955 (+ 5828us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:27.543005 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.543007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.544350 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.544351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.545608 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.545610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.546129 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.546129 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.546181 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.546561 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:27.546565 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:27.548359 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:27.550141 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.550142 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.550142 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.550324 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.550501 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.550502 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.550502 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.550557 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.550782 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:27.550783 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:27.550822 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 06:40:27.550886 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 06:40:27.550989 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":88,"threads_started":1}
I20260928 06:40:27.551626 26315 heartbeater.cc:347] Connected to a master server at 127.18.211.125:37143
I20260928 06:40:27.551681 26315 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:27.551755 26315 heartbeater.cc:511] Master 127.18.211.125:37143 requested a full tablet report, sending...
I20260928 06:40:27.552069 25948 ts_manager.cc:194] Registered new tserver with Master: 538426449b59422daa1419fc97acf5a6 (127.18.211.65:35203)
W20260928 06:40:27.625299 26321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:27.625501 26321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:27.625523 26321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:27.626652 26321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:27.626693 26321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:27.626718 26321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.66
I20260928 06:40:27.628334 26321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.66.keytab
--principal=oryx/127.18.211.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.66:44887
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.211.66
--webserver_port=34153
--tserver_master_addrs=127.18.211.126:44085,127.18.211.125:37143,127.18.211.124:40615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.66
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:27.628561 26321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:27.628786 26321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:27.633525 26326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:27.633498 26327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:27.633507 26329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:27.633718 26321 server_base.cc:1034] running on GCE node
I20260928 06:40:27.633951 26321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:27.634181 26321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:27.635318 26321 hybrid_clock.cc:648] HybridClock initialized: now 1790577627635304 us; error 30 us; skew 500 ppm
Sep 28 06:40:27 dist-test-slave-b28r krb5kdc[25306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577627, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:27.637822 26321 init.cc:377] Logged in from keytab as oryx/127.18.211.66@KRBTEST.COM (short username oryx)
I20260928 06:40:27.638257 26321 webserver.cc:460] Webserver started at http://127.18.211.66:34153/ using document root <none> and password file <none>
I20260928 06:40:27.638468 26321 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:27.638520 26321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:27.639808 26321 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:40:27.640305 26336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:27.640456 26321 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:27.640534 26321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-1/wal
uuid: "becf12ef2f814e66b83a1ddee4fa138b"
format_stamp: "Formatted at 2026-09-28 06:40:26 on dist-test-slave-b28r"
server_key: "dc4e69dfbffd56d3fee2206e8d44609b"
server_key_iv: "058b46256bf72a7cb2ce458c817628e0"
server_key_version: "encryptionkey@0"
I20260928 06:40:27.640954 26321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:27.651558 26321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:27.653501 26321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:27.653610 26321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:27.653805 26321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:27.654098 26321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:27.654132 26321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:27.654152 26321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:27.654163 26321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:27.659577 26321 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.66:44887
I20260928 06:40:27.659648 26449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.66:44887 every 8 connection(s)
I20260928 06:40:27.659964 26321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:40:27 dist-test-slave-b28r krb5kdc[25306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577627, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.125@KRBTEST.COM
Sep 28 06:40:27 dist-test-slave-b28r krb5kdc[25306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577627, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.124@KRBTEST.COM
Sep 28 06:40:27 dist-test-slave-b28r krb5kdc[25306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577627, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM
I20260928 06:40:27.668592 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 26321
I20260928 06:40:27.668744 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 25681
I20260928 06:40:27.671231 25999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.660744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:45571 (local address 127.18.211.125:37143)
0928 06:40:27.660786 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:27.660787 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:27.661115 (+ 328us) server_negotiation.cc:407] Connection header received
0928 06:40:27.662298 (+ 1183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:27.662299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:27.662311 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:27.662362 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:27.663060 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.663575 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.664840 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.665104 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.665154 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.667331 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:27.667333 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:27.667333 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:27.667340 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:27.669187 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.670119 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.670121 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.670121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.670178 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.670482 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.670483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.670484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.670632 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:27.670679 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:27.671032 (+ 353us) server_negotiation.cc:299] Negotiation successful
0928 06:40:27.671134 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:40:27.671254 26319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.660715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:53801 (local address 127.18.211.124:40615)
0928 06:40:27.660767 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:27.660769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:27.661087 (+ 318us) server_negotiation.cc:407] Connection header received
0928 06:40:27.662101 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:27.662103 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:27.662115 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:27.662178 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:27.663067 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.663834 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.665202 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.665458 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.665519 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.667775 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:27.667777 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:27.667778 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:27.667786 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:27.669871 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.670256 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.670257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.670258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.670312 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.670492 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.670492 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.670493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.670615 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:27.670657 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:27.671032 (+ 375us) server_negotiation.cc:299] Negotiation successful
0928 06:40:27.671147 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:40:27.671835 26454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.660704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:40615 (local address 127.18.211.66:53801)
0928 06:40:27.660947 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:27.660992 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:27.661990 (+ 998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:27.662249 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:27.662253 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:27.662707 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:27.662952 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.662958 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.663915 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.663916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.664755 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.664757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.665527 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.665528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.665592 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.665857 (+ 265us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:27.665862 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:27.667639 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:27.669952 (+ 2313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.669953 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.669953 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.670180 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.670377 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.670378 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.670378 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.670424 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.670703 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:27.670710 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:27.670878 (+ 168us) client_negotiation.cc:769] Sending connection context
0928 06:40:27.671399 (+ 521us) client_negotiation.cc:240] Negotiation successful
0928 06:40:27.671594 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":55,"thread_start_us":173,"threads_started":1}
I20260928 06:40:27.671838 26455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.660704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.125:37143 (local address 127.18.211.66:45571)
0928 06:40:27.660947 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:27.660992 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:27.662001 (+ 1009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:27.662380 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:27.662380 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:27.662717 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:27.662952 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.662958 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.663672 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.663672 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.664757 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.664758 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.665173 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.665174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.665228 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.665733 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:27.665738 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:27.667230 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:27.669754 (+ 2524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.669762 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.669764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.670039 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.670282 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.670283 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.670283 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.670409 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.670767 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:27.670768 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:27.670876 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 06:40:27.671399 (+ 523us) client_negotiation.cc:240] Negotiation successful
0928 06:40:27.671652 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":137,"thread_start_us":166,"threads_started":1}
I20260928 06:40:27.672274 26452 heartbeater.cc:347] Connected to a master server at 127.18.211.125:37143
I20260928 06:40:27.672371 26452 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:27.672580 26452 heartbeater.cc:511] Master 127.18.211.125:37143 requested a full tablet report, sending...
I20260928 06:40:27.672590 25409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.661257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:38533 (local address 127.18.211.126:44085)
0928 06:40:27.661300 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:27.661302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:27.661309 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:40:27.662176 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:27.662177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:27.662190 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:27.662237 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:27.663039 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.663841 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.664686 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.664910 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.664967 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.668233 (+ 3266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:27.668234 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:27.668235 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:27.668243 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:27.670576 (+ 2333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.670948 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.670949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.670949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.671005 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.671204 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.671205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.671205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.671347 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:27.671387 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:27.672425 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0928 06:40:27.672505 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:40:27.673107 26456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.661078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44085 (local address 127.18.211.66:38533)
0928 06:40:27.661195 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:27.661204 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:27.662016 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:27.662267 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:27.662268 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:27.662706 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:27.662951 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.662955 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.663943 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.663944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.664605 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.664606 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.664977 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.664978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.665074 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.665598 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:27.665608 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:27.668156 (+ 2548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:27.670664 (+ 2508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.670665 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.670666 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.670869 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.671065 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.671066 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.671066 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.671139 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.671806 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:27.671807 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:27.671851 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 06:40:27.672693 (+ 842us) client_negotiation.cc:240] Negotiation successful
0928 06:40:27.672805 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":23,"thread_start_us":65,"threads_started":1}
I20260928 06:40:27.673424 26451 heartbeater.cc:347] Connected to a master server at 127.18.211.126:44085
I20260928 06:40:27.673472 26451 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:27.673563 26451 heartbeater.cc:511] Master 127.18.211.126:44085 requested a full tablet report, sending...
I20260928 06:40:27.673841 25351 ts_manager.cc:194] Re-registered known tserver with Master: becf12ef2f814e66b83a1ddee4fa138b (127.18.211.66:44887)
I20260928 06:40:27.674024 25948 ts_manager.cc:194] Registered new tserver with Master: becf12ef2f814e66b83a1ddee4fa138b (127.18.211.66:44887)
I20260928 06:40:27.674342 25351 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.211.66@KRBTEST.COM'} at 127.18.211.66:38533
I20260928 06:40:27.675140 26450 heartbeater.cc:347] Connected to a master server at 127.18.211.124:40615
I20260928 06:40:27.675192 26450 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:27.675249 26450 heartbeater.cc:511] Master 127.18.211.124:40615 requested a full tablet report, sending...
I20260928 06:40:27.675531 26133 ts_manager.cc:194] Registered new tserver with Master: becf12ef2f814e66b83a1ddee4fa138b (127.18.211.66:44887)
I20260928 06:40:27.675815 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.67:39055
--local_ip_for_outbound_sockets=127.18.211.67
--tserver_master_addrs=127.18.211.126:44085
--webserver_port=41715
--webserver_interface=127.18.211.67
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.67.keytab
--principal=oryx/127.18.211.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.18.211.126:44085,127.18.211.125:37143
--tserver_master_addrs=127.18.211.126:44085,127.18.211.125:37143,127.18.211.124:40615 with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:27.754971 26457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:27.755216 26457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:27.755244 26457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:27.756357 26457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:27.756389 26457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:27.756413 26457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.67
I20260928 06:40:27.757790 26457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.67.keytab
--principal=oryx/127.18.211.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.67:39055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.211.67
--webserver_port=41715
--tserver_master_addrs=127.18.211.126:44085,127.18.211.125:37143,127.18.211.124:40615
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.67
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:27.757970 26457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:27.758147 26457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:27.762955 26465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:27.763130 26457 server_base.cc:1034] running on GCE node
W20260928 06:40:27.762965 26463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:27.763000 26462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:27.763432 26457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:27.763649 26457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:27.764796 26457 hybrid_clock.cc:648] HybridClock initialized: now 1790577627764785 us; error 33 us; skew 500 ppm
Sep 28 06:40:27 dist-test-slave-b28r krb5kdc[25306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577627, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:27.767345 26457 init.cc:377] Logged in from keytab as oryx/127.18.211.67@KRBTEST.COM (short username oryx)
I20260928 06:40:27.767764 26457 webserver.cc:460] Webserver started at http://127.18.211.67:41715/ using document root <none> and password file <none>
I20260928 06:40:27.767987 26457 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:27.768049 26457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:27.769381 26457 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:40:27.769992 26472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:27.770138 26457 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:40:27.770231 26457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-2/wal
uuid: "a25c0921c520499aa06c828f01791c1d"
format_stamp: "Formatted at 2026-09-28 06:40:26 on dist-test-slave-b28r"
server_key: "410455ad97777eb86fade27764db050e"
server_key_iv: "9b0dd349c27cd84662369fb26560e962"
server_key_version: "encryptionkey@0"
I20260928 06:40:27.770648 26457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:27.773995 25908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.770994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:40615 (local address 127.0.0.1:58658)
0928 06:40:27.771121 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:27.771132 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:27.771189 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:27.771346 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:27.771347 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:27.771348 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:27.771400 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.771402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.772319 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.772319 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.773024 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.773027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.773808 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.773809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.773861 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.773865 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:40:27.773908 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 06:40:27.773942 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260928 06:40:27.774014 26319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.771055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58658 (local address 127.18.211.124:40615)
0928 06:40:27.771126 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:27.771129 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:27.771176 (+ 47us) server_negotiation.cc:407] Connection header received
0928 06:40:27.771237 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:27.771238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:27.771264 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:27.771344 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:27.771439 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.772215 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.773114 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.773723 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.773798 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.773867 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:27.773914 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 06:40:27.773949 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 06:40:27.774367 26143 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 06:40:27.774576 26173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f5d90cb26f1a4900a3e9a927bb0d79be. Latest consensus state: current_term: 1 leader_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: false } } peers { permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: NON_VOTER last_known_addr { host: "127.18.211.124" port: 40615 } attrs { promote: true } } }
I20260928 06:40:27.774672 26173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [sys.catalog]: This master's current role is: LEARNER
I20260928 06:40:27.783145 26457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:27.784911 26457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:27.785012 26457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:27.785182 26457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:27.785429 26457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:27.785456 26457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:27.785476 26457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:27.785487 26457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:27.790755 26457 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.67:39055
I20260928 06:40:27.790838 26585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.67:39055 every 8 connection(s)
I20260928 06:40:27.791170 26457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:40:27 dist-test-slave-b28r krb5kdc[25306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577627, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.124@KRBTEST.COM
Sep 28 06:40:27 dist-test-slave-b28r krb5kdc[25306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577627, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.125@KRBTEST.COM
Sep 28 06:40:27 dist-test-slave-b28r krb5kdc[25306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577627, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM
I20260928 06:40:27.800287 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 26457
I20260928 06:40:27.801277 25999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.791880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:48935 (local address 127.18.211.125:37143)
0928 06:40:27.791930 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:27.791932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:27.792239 (+ 307us) server_negotiation.cc:407] Connection header received
0928 06:40:27.792805 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:27.792806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:27.792819 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:27.792850 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:27.793561 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.794303 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.795380 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.795643 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.795697 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.798244 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:27.798245 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:27.798245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:27.798267 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:27.799866 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.800311 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.800312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.800312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.800377 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.800596 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.800597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.800598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.800715 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:27.800774 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:27.801091 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 06:40:27.801173 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:40:27.801792 26592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.792148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44085 (local address 127.18.211.67:46415)
0928 06:40:27.792260 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:27.792268 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:27.792686 (+ 418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:27.792917 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:27.792918 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:27.793301 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:27.793444 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.793450 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.794155 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.794156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.795317 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.795319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.795797 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.795798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.795854 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.796433 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:27.796436 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:27.798777 (+ 2341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:27.800487 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.800488 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.800488 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.800684 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.800891 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.800892 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.800892 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.800950 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.801246 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:27.801247 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:27.801300 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 06:40:27.801397 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 06:40:27.801563 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":7,"thread_start_us":65,"threads_started":1}
I20260928 06:40:27.801792 26590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.791880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.125:37143 (local address 127.18.211.67:48935)
0928 06:40:27.792169 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:27.792179 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:27.792686 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:27.792919 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:27.792919 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:27.793375 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:27.793443 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.793450 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.794386 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.794387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.795291 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.795293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.795707 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.795708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.795778 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.796238 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:27.796248 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:27.798143 (+ 1895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:27.799963 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.799964 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.799964 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.800232 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.800426 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.800427 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.800427 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.800538 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.800780 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:27.800787 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:27.800936 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 06:40:27.801584 (+ 648us) client_negotiation.cc:240] Negotiation successful
0928 06:40:27.801692 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":25,"thread_start_us":89,"threads_started":1}
I20260928 06:40:27.802361 26588 heartbeater.cc:347] Connected to a master server at 127.18.211.125:37143
I20260928 06:40:27.802451 26588 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:27.802596 25409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.792332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:46415 (local address 127.18.211.126:44085)
0928 06:40:27.792382 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:27.792385 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:27.792393 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:40:27.792804 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:27.792806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:27.792818 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:27.792850 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:27.793550 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.794057 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.795408 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.795670 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.795744 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.798883 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:27.798884 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:27.798885 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:27.798893 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:27.800399 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.800761 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.800762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.800762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.800816 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.801034 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.801036 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.801036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.801175 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:27.801240 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:27.802441 (+ 1201us) server_negotiation.cc:299] Negotiation successful
0928 06:40:27.802517 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:40:27.802644 26588 heartbeater.cc:511] Master 127.18.211.125:37143 requested a full tablet report, sending...
I20260928 06:40:27.802990 26587 heartbeater.cc:347] Connected to a master server at 127.18.211.126:44085
I20260928 06:40:27.803040 26587 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:27.803133 26587 heartbeater.cc:511] Master 127.18.211.126:44085 requested a full tablet report, sending...
I20260928 06:40:27.803133 25948 ts_manager.cc:194] Registered new tserver with Master: a25c0921c520499aa06c828f01791c1d (127.18.211.67:39055)
I20260928 06:40:27.803289 26319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.791882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:55735 (local address 127.18.211.124:40615)
0928 06:40:27.792080 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:27.792082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:27.792167 (+ 85us) server_negotiation.cc:407] Connection header received
0928 06:40:27.792762 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:27.792763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:27.792775 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:27.792825 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:27.793532 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.794322 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.795229 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.795467 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.795527 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.797884 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:27.797886 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:27.797887 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:27.797895 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:27.799759 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.801894 (+ 2135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.801895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.801895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.801957 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.802154 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.802156 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.802156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.802309 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:27.802372 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:27.803120 (+ 748us) server_negotiation.cc:299] Negotiation successful
0928 06:40:27.803202 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260928 06:40:27.803493 25351 ts_manager.cc:194] Re-registered known tserver with Master: a25c0921c520499aa06c828f01791c1d (127.18.211.67:39055)
I20260928 06:40:27.803737 26591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.791878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:40615 (local address 127.18.211.67:55735)
0928 06:40:27.792046 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:27.792086 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:27.792660 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:27.792897 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:27.792901 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:27.793240 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:27.793442 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.793447 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.794402 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.794403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.795148 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.795151 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.795539 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.795539 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.795682 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.796331 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:27.796337 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:27.797796 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:27.799879 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.799887 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.799889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.801472 (+ 1583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.802031 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:27.802032 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:27.802032 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:27.802086 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:27.802820 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:27.802821 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:27.802861 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 06:40:27.802931 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 06:40:27.803022 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":19,"thread_start_us":89,"threads_started":1}
I20260928 06:40:27.803994 25351 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.211.67@KRBTEST.COM'} at 127.18.211.67:46415
I20260928 06:40:27.804363 26586 heartbeater.cc:347] Connected to a master server at 127.18.211.124:40615
I20260928 06:40:27.804409 26586 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:27.804461 26586 heartbeater.cc:511] Master 127.18.211.124:40615 requested a full tablet report, sending...
I20260928 06:40:27.804718 26133 ts_manager.cc:194] Registered new tserver with Master: a25c0921c520499aa06c828f01791c1d (127.18.211.67:39055)
I20260928 06:40:27.809548 25999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.801933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56782 (local address 127.18.211.125:37143)
0928 06:40:27.802291 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:27.802293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:27.802301 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:40:27.802319 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:27.802320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:27.802328 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:27.802379 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:27.803451 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.804161 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.805212 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.805437 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.805490 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.806899 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:27.806900 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:27.806900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:27.806907 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:27.808367 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.808723 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.808724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.808724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.808794 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.809004 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.809005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.809005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.809133 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:27.809222 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:27.809389 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 06:40:27.809457 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324}
I20260928 06:40:27.811240 25409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.802154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43484 (local address 127.18.211.126:44085)
0928 06:40:27.802670 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:27.802672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:27.803572 (+ 900us) server_negotiation.cc:407] Connection header received
0928 06:40:27.803665 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:27.803666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:27.803675 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:27.803701 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:27.804385 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.805094 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.805932 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.806198 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.806253 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.808307 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:27.808308 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:27.808308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:27.808315 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:27.810179 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.810554 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.810555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.810555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.810606 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.810783 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.810784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.810784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.810887 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:27.810971 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:27.811074 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 06:40:27.811154 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":85,"threads_started":1}
I20260928 06:40:27.813001 26599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.801876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58670 (local address 127.18.211.124:40615)
0928 06:40:27.802571 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:27.802573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:27.806040 (+ 3467us) server_negotiation.cc:407] Connection header received
0928 06:40:27.806067 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:27.806068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:27.806080 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:27.806142 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:27.806709 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.807387 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.808152 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.808381 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.808436 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.810358 (+ 1922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:27.810360 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:27.810360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:27.810367 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:27.812025 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.812316 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.812317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.812317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.812374 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:27.812532 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:27.812533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:27.812533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:27.812654 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:27.812755 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:27.812839 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 06:40:27.812914 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":66,"threads_started":1}
I20260928 06:40:27.812733 25351 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43484:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:40:27.814608 25351 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:40:27.815321 25909 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: false } } peers { permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: NON_VOTER last_known_addr { host: "127.18.211.124" port: 40615 } attrs { promote: true } } }
I20260928 06:40:27.815420 25909 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:27.816800 26095 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be: attempting to promote NON_VOTER 90b91ba24e8c4c20853a01477d44c493 to VOTER
I20260928 06:40:27.816932 26095 consensus_queue.cc:237] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: false } } peers { permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 40615 } attrs { promote: false } }
I20260928 06:40:27.817308 26143 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [term 1 LEARNER]: Refusing update from remote peer f5d90cb26f1a4900a3e9a927bb0d79be: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 06:40:27.817410 25958 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [term 1 FOLLOWER]: Refusing update from remote peer f5d90cb26f1a4900a3e9a927bb0d79be: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 06:40:27.817482 25910 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [LEADER]: Connected to new peer: Peer: permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 40615 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260928 06:40:27.817577 26603 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [LEADER]: Connected to new peer: Peer: permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260928 06:40:27.818257 26095 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 90b91ba24e8c4c20853a01477d44c493 (127.18.211.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: false } } peers { permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 40615 } attrs { promote: false } } }
I20260928 06:40:27.818428 25958 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 90b91ba24e8c4c20853a01477d44c493 (127.18.211.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: false } } peers { permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 40615 } attrs { promote: false } } }
I20260928 06:40:27.818392 26143 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 90b91ba24e8c4c20853a01477d44c493 (127.18.211.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: false } } peers { permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 40615 } attrs { promote: false } } }
I20260928 06:40:27.819247 26095 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: false } } peers { permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 40615 } attrs { promote: false } } }
I20260928 06:40:27.819326 26095 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:27.819375 26095 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: false } } peers { permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 40615 } attrs { promote: false } } }
I20260928 06:40:27.819415 26173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: false } } peers { permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 40615 } attrs { promote: false } } }
I20260928 06:40:27.819415 26603 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: false } } peers { permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 40615 } attrs { promote: false } } }
I20260928 06:40:27.819491 26173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:40:27.819492 26603 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:27.819514 26098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: false } } peers { permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 40615 } attrs { promote: false } } }
I20260928 06:40:27.819588 26098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:40:27.819422 26095 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:27.824003 26612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.820359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:44887 (local address 127.0.0.1:49618)
0928 06:40:27.820457 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:27.820466 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:27.820517 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:27.820914 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:27.820915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:27.820917 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:27.820955 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.820956 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.822111 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.822112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.822857 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.822859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.823786 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.823787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.823851 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.823856 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:40:27.823887 (+ 31us) client_negotiation.cc:240] Negotiation successful
0928 06:40:27.823927 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":3,"thread_start_us":65,"threads_started":1}
I20260928 06:40:27.824110 26610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.820154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:39055 (local address 127.0.0.1:43736)
0928 06:40:27.820314 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:27.820323 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:27.820369 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:27.820815 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:27.820816 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:27.820818 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:27.820864 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.820866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.821937 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.821938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.823154 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.823156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.823893 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.823894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.823956 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.823960 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:40:27.823990 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 06:40:27.824028 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":85,"thread_start_us":82,"threads_started":1}
I20260928 06:40:27.824333 25908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.820153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:35203 (local address 127.0.0.1:38940)
0928 06:40:27.820204 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:27.820215 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:27.820260 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:27.820622 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:27.820623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:27.820625 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:27.820672 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.820674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.822000 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.822001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.823135 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:27.823138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.823974 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.823975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:27.824027 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.824031 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:40:27.824059 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 06:40:27.824095 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":6}
I20260928 06:40:27.824401 26614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.820559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49618 (local address 127.18.211.66:44887)
0928 06:40:27.820748 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:27.820752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:27.820762 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:40:27.820806 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:27.820808 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:27.820823 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:27.820894 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:27.821018 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.822032 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.822937 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.823674 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.824158 (+ 484us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.824259 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:27.824287 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 06:40:27.824326 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":84,"threads_started":1}
I20260928 06:40:27.824523 26613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.820479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43736 (local address 127.18.211.67:39055)
0928 06:40:27.820659 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:27.820664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:27.820673 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:40:27.820712 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:27.820714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:27.820729 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:27.820804 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:27.820941 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.821870 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.823234 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.823741 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.824278 (+ 537us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.824374 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:27.824402 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 06:40:27.824449 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":99,"threads_started":1}
I20260928 06:40:27.824728 26611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:27.820275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38940 (local address 127.18.211.65:35203)
0928 06:40:27.820443 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:27.820448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:27.820459 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:40:27.820498 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:27.820500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:27.820519 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:27.820750 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:27.820815 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.821926 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.823206 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:27.823893 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:27.824283 (+ 390us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:27.824367 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:27.824598 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 06:40:27.824645 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":97,"threads_started":1}
I20260928 06:40:27.825068 26520 tablet_service.cc:1467] Processing CreateTablet for tablet 93c9e4253d5844b5a89569e2035537ba (DEFAULT_TABLE table=test-table [id=d305f2f5db23465dbbf45019799302b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:27.825066 26384 tablet_service.cc:1467] Processing CreateTablet for tablet 93c9e4253d5844b5a89569e2035537ba (DEFAULT_TABLE table=test-table [id=d305f2f5db23465dbbf45019799302b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:27.825371 26384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93c9e4253d5844b5a89569e2035537ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:27.825371 26520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93c9e4253d5844b5a89569e2035537ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:27.827764 26247 tablet_service.cc:1467] Processing CreateTablet for tablet 93c9e4253d5844b5a89569e2035537ba (DEFAULT_TABLE table=test-table [id=d305f2f5db23465dbbf45019799302b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:27.828116 26247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93c9e4253d5844b5a89569e2035537ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:27.828244 26615 tablet_bootstrap.cc:492] T 93c9e4253d5844b5a89569e2035537ba P a25c0921c520499aa06c828f01791c1d: Bootstrap starting.
I20260928 06:40:27.828892 26615 tablet_bootstrap.cc:654] T 93c9e4253d5844b5a89569e2035537ba P a25c0921c520499aa06c828f01791c1d: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:27.829106 26615 log.cc:824] T 93c9e4253d5844b5a89569e2035537ba P a25c0921c520499aa06c828f01791c1d: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:27.829391 26616 tablet_bootstrap.cc:492] T 93c9e4253d5844b5a89569e2035537ba P becf12ef2f814e66b83a1ddee4fa138b: Bootstrap starting.
I20260928 06:40:27.829861 26615 tablet_bootstrap.cc:492] T 93c9e4253d5844b5a89569e2035537ba P a25c0921c520499aa06c828f01791c1d: No bootstrap required, opened a new log
I20260928 06:40:27.829890 26616 tablet_bootstrap.cc:654] T 93c9e4253d5844b5a89569e2035537ba P becf12ef2f814e66b83a1ddee4fa138b: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:27.829921 26615 ts_tablet_manager.cc:1397] T 93c9e4253d5844b5a89569e2035537ba P a25c0921c520499aa06c828f01791c1d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:40:27.830096 26616 log.cc:824] T 93c9e4253d5844b5a89569e2035537ba P becf12ef2f814e66b83a1ddee4fa138b: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:27.830701 26616 tablet_bootstrap.cc:492] T 93c9e4253d5844b5a89569e2035537ba P becf12ef2f814e66b83a1ddee4fa138b: No bootstrap required, opened a new log
I20260928 06:40:27.830742 26616 ts_tablet_manager.cc:1397] T 93c9e4253d5844b5a89569e2035537ba P becf12ef2f814e66b83a1ddee4fa138b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:27.831951 26615 raft_consensus.cc:348] T 93c9e4253d5844b5a89569e2035537ba P a25c0921c520499aa06c828f01791c1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538426449b59422daa1419fc97acf5a6" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35203 } } peers { permanent_uuid: "a25c0921c520499aa06c828f01791c1d" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 39055 } } peers { permanent_uuid: "becf12ef2f814e66b83a1ddee4fa138b" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 44887 } }
I20260928 06:40:27.832108 26615 raft_consensus.cc:374] T 93c9e4253d5844b5a89569e2035537ba P a25c0921c520499aa06c828f01791c1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:27.832144 26615 raft_consensus.cc:729] T 93c9e4253d5844b5a89569e2035537ba P a25c0921c520499aa06c828f01791c1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a25c0921c520499aa06c828f01791c1d, State: Initialized, Role: FOLLOWER
I20260928 06:40:27.832161 26619 tablet_bootstrap.cc:492] T 93c9e4253d5844b5a89569e2035537ba P 538426449b59422daa1419fc97acf5a6: Bootstrap starting.
I20260928 06:40:27.832271 26615 consensus_queue.cc:260] T 93c9e4253d5844b5a89569e2035537ba P a25c0921c520499aa06c828f01791c1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538426449b59422daa1419fc97acf5a6" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35203 } } peers { permanent_uuid: "a25c0921c520499aa06c828f01791c1d" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 39055 } } peers { permanent_uuid: "becf12ef2f814e66b83a1ddee4fa138b" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 44887 } }
I20260928 06:40:27.832515 26615 ts_tablet_manager.cc:1428] T 93c9e4253d5844b5a89569e2035537ba P a25c0921c520499aa06c828f01791c1d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 06:40:27.832634 26587 heartbeater.cc:503] Master 127.18.211.126:44085 was elected leader, sending a full tablet report...
I20260928 06:40:27.832927 26619 tablet_bootstrap.cc:654] T 93c9e4253d5844b5a89569e2035537ba P 538426449b59422daa1419fc97acf5a6: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:27.832912 26616 raft_consensus.cc:348] T 93c9e4253d5844b5a89569e2035537ba P becf12ef2f814e66b83a1ddee4fa138b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538426449b59422daa1419fc97acf5a6" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35203 } } peers { permanent_uuid: "a25c0921c520499aa06c828f01791c1d" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 39055 } } peers { permanent_uuid: "becf12ef2f814e66b83a1ddee4fa138b" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 44887 } }
I20260928 06:40:27.833074 26616 raft_consensus.cc:374] T 93c9e4253d5844b5a89569e2035537ba P becf12ef2f814e66b83a1ddee4fa138b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:27.833107 26616 raft_consensus.cc:729] T 93c9e4253d5844b5a89569e2035537ba P becf12ef2f814e66b83a1ddee4fa138b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: becf12ef2f814e66b83a1ddee4fa138b, State: Initialized, Role: FOLLOWER
I20260928 06:40:27.833211 26616 consensus_queue.cc:260] T 93c9e4253d5844b5a89569e2035537ba P becf12ef2f814e66b83a1ddee4fa138b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538426449b59422daa1419fc97acf5a6" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35203 } } peers { permanent_uuid: "a25c0921c520499aa06c828f01791c1d" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 39055 } } peers { permanent_uuid: "becf12ef2f814e66b83a1ddee4fa138b" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 44887 } }
I20260928 06:40:27.833240 26619 log.cc:824] T 93c9e4253d5844b5a89569e2035537ba P 538426449b59422daa1419fc97acf5a6: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:27.833467 26616 ts_tablet_manager.cc:1428] T 93c9e4253d5844b5a89569e2035537ba P becf12ef2f814e66b83a1ddee4fa138b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:40:27.833535 26451 heartbeater.cc:503] Master 127.18.211.126:44085 was elected leader, sending a full tablet report...
I20260928 06:40:27.834241 26619 tablet_bootstrap.cc:492] T 93c9e4253d5844b5a89569e2035537ba P 538426449b59422daa1419fc97acf5a6: No bootstrap required, opened a new log
I20260928 06:40:27.834306 26619 ts_tablet_manager.cc:1397] T 93c9e4253d5844b5a89569e2035537ba P 538426449b59422daa1419fc97acf5a6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:40:27.836473 26619 raft_consensus.cc:348] T 93c9e4253d5844b5a89569e2035537ba P 538426449b59422daa1419fc97acf5a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538426449b59422daa1419fc97acf5a6" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35203 } } peers { permanent_uuid: "a25c0921c520499aa06c828f01791c1d" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 39055 } } peers { permanent_uuid: "becf12ef2f814e66b83a1ddee4fa138b" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 44887 } }
I20260928 06:40:27.836592 26619 raft_consensus.cc:374] T 93c9e4253d5844b5a89569e2035537ba P 538426449b59422daa1419fc97acf5a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:27.836612 26619 raft_consensus.cc:729] T 93c9e4253d5844b5a89569e2035537ba P 538426449b59422daa1419fc97acf5a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 538426449b59422daa1419fc97acf5a6, State: Initialized, Role: FOLLOWER
I20260928 06:40:27.836724 26619 consensus_queue.cc:260] T 93c9e4253d5844b5a89569e2035537ba P 538426449b59422daa1419fc97acf5a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538426449b59422daa1419fc97acf5a6" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35203 } } peers { permanent_uuid: "a25c0921c520499aa06c828f01791c1d" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 39055 } } peers { permanent_uuid: "becf12ef2f814e66b83a1ddee4fa138b" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 44887 } }
I20260928 06:40:27.836935 26619 ts_tablet_manager.cc:1428] T 93c9e4253d5844b5a89569e2035537ba P 538426449b59422daa1419fc97acf5a6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:40:27.837018 26314 heartbeater.cc:503] Master 127.18.211.126:44085 was elected leader, sending a full tablet report...
W20260928 06:40:27.911046 26453 tablet.cc:2369] T 93c9e4253d5844b5a89569e2035537ba P becf12ef2f814e66b83a1ddee4fa138b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:40:28.036477 26316 tablet.cc:2369] T 93c9e4253d5844b5a89569e2035537ba P 538426449b59422daa1419fc97acf5a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:40:28.041991 26589 tablet.cc:2369] T 93c9e4253d5844b5a89569e2035537ba P a25c0921c520499aa06c828f01791c1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:40:28.113411 26620 raft_consensus.cc:482] T 93c9e4253d5844b5a89569e2035537ba P a25c0921c520499aa06c828f01791c1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:40:28.113559 26620 raft_consensus.cc:504] T 93c9e4253d5844b5a89569e2035537ba P a25c0921c520499aa06c828f01791c1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538426449b59422daa1419fc97acf5a6" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35203 } } peers { permanent_uuid: "a25c0921c520499aa06c828f01791c1d" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 39055 } } peers { permanent_uuid: "becf12ef2f814e66b83a1ddee4fa138b" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 44887 } }
I20260928 06:40:28.113938 26620 leader_election.cc:288] T 93c9e4253d5844b5a89569e2035537ba P a25c0921c520499aa06c828f01791c1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 538426449b59422daa1419fc97acf5a6 (127.18.211.65:35203), becf12ef2f814e66b83a1ddee4fa138b (127.18.211.66:44887)
I20260928 06:40:28.117723 26611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:28.114241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:35069 (local address 127.18.211.65:35203)
0928 06:40:28.114285 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:28.114287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:28.114297 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:40:28.114327 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:28.114329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:28.114338 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:28.114404 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:28.114607 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:28.115387 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:28.116826 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:28.117391 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:28.117450 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:28.117457 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:28.117617 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 06:40:28.117652 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:40:28.117724 26614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:28.114217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:38653 (local address 127.18.211.66:44887)
0928 06:40:28.114317 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:28.114319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:28.114326 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:40:28.114349 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:28.114350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:28.114358 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:28.114408 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:28.114612 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:28.115381 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:28.116826 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:28.117373 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:28.117449 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:28.117457 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:28.117617 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 06:40:28.117654 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 06:40:28.117770 26590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:28.114110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:35203 (local address 127.18.211.67:35069)
0928 06:40:28.114230 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:28.114239 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:28.114273 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:28.114425 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:28.114426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:28.114427 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:28.114547 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:28.114548 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:28.115467 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:28.115468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:28.116739 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:28.116741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:28.117458 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:28.117459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:28.117545 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:28.117550 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:40:28.117608 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 06:40:28.117648 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":147}
I20260928 06:40:28.117782 26591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:28.114110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:44887 (local address 127.18.211.67:38653)
0928 06:40:28.114178 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:28.114191 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:28.114253 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:28.114417 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:28.114418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:28.114420 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:28.114539 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:28.114541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:28.115467 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:28.115468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:28.116739 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:28.116742 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:28.117458 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:28.117459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:28.117551 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:28.117553 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:40:28.117608 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 06:40:28.117647 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":75}
I20260928 06:40:28.118103 26267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93c9e4253d5844b5a89569e2035537ba" candidate_uuid: "a25c0921c520499aa06c828f01791c1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "538426449b59422daa1419fc97acf5a6" is_pre_election: true
I20260928 06:40:28.118108 26404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93c9e4253d5844b5a89569e2035537ba" candidate_uuid: "a25c0921c520499aa06c828f01791c1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "becf12ef2f814e66b83a1ddee4fa138b" is_pre_election: true
I20260928 06:40:28.118258 26404 raft_consensus.cc:2454] T 93c9e4253d5844b5a89569e2035537ba P becf12ef2f814e66b83a1ddee4fa138b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a25c0921c520499aa06c828f01791c1d in term 0.
I20260928 06:40:28.118258 26267 raft_consensus.cc:2454] T 93c9e4253d5844b5a89569e2035537ba P 538426449b59422daa1419fc97acf5a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a25c0921c520499aa06c828f01791c1d in term 0.
I20260928 06:40:28.118494 26474 leader_election.cc:302] T 93c9e4253d5844b5a89569e2035537ba P a25c0921c520499aa06c828f01791c1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a25c0921c520499aa06c828f01791c1d, becf12ef2f814e66b83a1ddee4fa138b; no voters:
I20260928 06:40:28.118650 26620 raft_consensus.cc:2781] T 93c9e4253d5844b5a89569e2035537ba P a25c0921c520499aa06c828f01791c1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:40:28.118709 26620 raft_consensus.cc:482] T 93c9e4253d5844b5a89569e2035537ba P a25c0921c520499aa06c828f01791c1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:40:28.118731 26620 raft_consensus.cc:3037] T 93c9e4253d5844b5a89569e2035537ba P a25c0921c520499aa06c828f01791c1d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:28.119513 26620 raft_consensus.cc:504] T 93c9e4253d5844b5a89569e2035537ba P a25c0921c520499aa06c828f01791c1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538426449b59422daa1419fc97acf5a6" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35203 } } peers { permanent_uuid: "a25c0921c520499aa06c828f01791c1d" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 39055 } } peers { permanent_uuid: "becf12ef2f814e66b83a1ddee4fa138b" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 44887 } }
I20260928 06:40:28.119639 26620 leader_election.cc:288] T 93c9e4253d5844b5a89569e2035537ba P a25c0921c520499aa06c828f01791c1d [CANDIDATE]: Term 1 election: Requested vote from peers 538426449b59422daa1419fc97acf5a6 (127.18.211.65:35203), becf12ef2f814e66b83a1ddee4fa138b (127.18.211.66:44887)
I20260928 06:40:28.119814 26404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93c9e4253d5844b5a89569e2035537ba" candidate_uuid: "a25c0921c520499aa06c828f01791c1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "becf12ef2f814e66b83a1ddee4fa138b"
I20260928 06:40:28.119843 26267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93c9e4253d5844b5a89569e2035537ba" candidate_uuid: "a25c0921c520499aa06c828f01791c1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "538426449b59422daa1419fc97acf5a6"
I20260928 06:40:28.119910 26267 raft_consensus.cc:3037] T 93c9e4253d5844b5a89569e2035537ba P 538426449b59422daa1419fc97acf5a6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:28.119907 26404 raft_consensus.cc:3037] T 93c9e4253d5844b5a89569e2035537ba P becf12ef2f814e66b83a1ddee4fa138b [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:28.120816 26267 raft_consensus.cc:2454] T 93c9e4253d5844b5a89569e2035537ba P 538426449b59422daa1419fc97acf5a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a25c0921c520499aa06c828f01791c1d in term 1.
I20260928 06:40:28.120816 26404 raft_consensus.cc:2454] T 93c9e4253d5844b5a89569e2035537ba P becf12ef2f814e66b83a1ddee4fa138b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a25c0921c520499aa06c828f01791c1d in term 1.
I20260928 06:40:28.121001 26474 leader_election.cc:302] T 93c9e4253d5844b5a89569e2035537ba P a25c0921c520499aa06c828f01791c1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a25c0921c520499aa06c828f01791c1d, becf12ef2f814e66b83a1ddee4fa138b; no voters:
I20260928 06:40:28.121081 26620 raft_consensus.cc:2781] T 93c9e4253d5844b5a89569e2035537ba P a25c0921c520499aa06c828f01791c1d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:28.121213 26620 raft_consensus.cc:686] T 93c9e4253d5844b5a89569e2035537ba P a25c0921c520499aa06c828f01791c1d [term 1 LEADER]: Becoming Leader. State: Replica: a25c0921c520499aa06c828f01791c1d, State: Running, Role: LEADER
I20260928 06:40:28.121366 26620 consensus_queue.cc:237] T 93c9e4253d5844b5a89569e2035537ba P a25c0921c520499aa06c828f01791c1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538426449b59422daa1419fc97acf5a6" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35203 } } peers { permanent_uuid: "a25c0921c520499aa06c828f01791c1d" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 39055 } } peers { permanent_uuid: "becf12ef2f814e66b83a1ddee4fa138b" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 44887 } }
I20260928 06:40:28.122253 25351 catalog_manager.cc:5495] T 93c9e4253d5844b5a89569e2035537ba P a25c0921c520499aa06c828f01791c1d reported cstate change: term changed from 0 to 1, leader changed from <none> to a25c0921c520499aa06c828f01791c1d (127.18.211.67). New cstate: current_term: 1 leader_uuid: "a25c0921c520499aa06c828f01791c1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "538426449b59422daa1419fc97acf5a6" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a25c0921c520499aa06c828f01791c1d" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 39055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "becf12ef2f814e66b83a1ddee4fa138b" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 44887 } health_report { overall_health: UNKNOWN } } }
I20260928 06:40:28.181375 26613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:28.179139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43744 (local address 127.18.211.67:39055)
0928 06:40:28.179192 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:28.179193 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:28.179200 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:40:28.179238 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:28.179239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:28.179257 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:28.179354 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:40:28.179477 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:28.179941 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:28.180688 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:28.180882 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:28.180939 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:28.181112 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:40:28.181209 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:40:28.181263 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:28.181300 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 06:40:28.181318 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:40:28.183175 26267 raft_consensus.cc:1261] T 93c9e4253d5844b5a89569e2035537ba P 538426449b59422daa1419fc97acf5a6 [term 1 FOLLOWER]: Refusing update from remote peer a25c0921c520499aa06c828f01791c1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:40:28.183193 26404 raft_consensus.cc:1261] T 93c9e4253d5844b5a89569e2035537ba P becf12ef2f814e66b83a1ddee4fa138b [term 1 FOLLOWER]: Refusing update from remote peer a25c0921c520499aa06c828f01791c1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:40:28.183514 26620 consensus_queue.cc:1035] T 93c9e4253d5844b5a89569e2035537ba P a25c0921c520499aa06c828f01791c1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "becf12ef2f814e66b83a1ddee4fa138b" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 44887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:40:28.183650 26624 consensus_queue.cc:1035] T 93c9e4253d5844b5a89569e2035537ba P a25c0921c520499aa06c828f01791c1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "538426449b59422daa1419fc97acf5a6" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:40:28.186393 26629 mvcc.cc:205] Tried to move back new op lower bound from 7334205965035593728 to 7334205964785639424. Current Snapshot: MvccSnapshot[applied={T|T < 7334205965035593728}]
I20260928 06:40:28.188885 25350 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:40:28.188970 25350 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:40:28.190481 25350 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f5d90cb26f1a4900a3e9a927bb0d79be: Sending DeleteTablet for 3 replicas of tablet 93c9e4253d5844b5a89569e2035537ba
I20260928 06:40:28.190794 26247 tablet_service.cc:1514] Processing DeleteTablet for tablet 93c9e4253d5844b5a89569e2035537ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:28 UTC) from {username='oryx', principal='oryx/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:38940
I20260928 06:40:28.190778 26520 tablet_service.cc:1514] Processing DeleteTablet for tablet 93c9e4253d5844b5a89569e2035537ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:28 UTC) from {username='oryx', principal='oryx/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:43736
I20260928 06:40:28.190858 26384 tablet_service.cc:1514] Processing DeleteTablet for tablet 93c9e4253d5844b5a89569e2035537ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:28 UTC) from {username='oryx', principal='oryx/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:49618
I20260928 06:40:28.190961 26637 tablet_replica.cc:330] T 93c9e4253d5844b5a89569e2035537ba P 538426449b59422daa1419fc97acf5a6: stopping tablet replica
I20260928 06:40:28.190991 26638 tablet_replica.cc:330] T 93c9e4253d5844b5a89569e2035537ba P a25c0921c520499aa06c828f01791c1d: stopping tablet replica
I20260928 06:40:28.191051 26637 raft_consensus.cc:2229] T 93c9e4253d5844b5a89569e2035537ba P 538426449b59422daa1419fc97acf5a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:40:28.191057 26638 raft_consensus.cc:2229] T 93c9e4253d5844b5a89569e2035537ba P a25c0921c520499aa06c828f01791c1d [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:40:28.191164 26638 raft_consensus.cc:2258] T 93c9e4253d5844b5a89569e2035537ba P a25c0921c520499aa06c828f01791c1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:28.191169 26637 raft_consensus.cc:2258] T 93c9e4253d5844b5a89569e2035537ba P 538426449b59422daa1419fc97acf5a6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:28.191813 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 25319
I20260928 06:40:28.192715 26637 ts_tablet_manager.cc:1905] T 93c9e4253d5844b5a89569e2035537ba P 538426449b59422daa1419fc97acf5a6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:28.194339 26637 ts_tablet_manager.cc:1918] T 93c9e4253d5844b5a89569e2035537ba P 538426449b59422daa1419fc97acf5a6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:40:28.194408 26637 log.cc:1196] T 93c9e4253d5844b5a89569e2035537ba P 538426449b59422daa1419fc97acf5a6: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-0/wal/wals/93c9e4253d5844b5a89569e2035537ba
I20260928 06:40:28.194643 26637 ts_tablet_manager.cc:1939] T 93c9e4253d5844b5a89569e2035537ba P 538426449b59422daa1419fc97acf5a6: Deleting consensus metadata
I20260928 06:40:28.195119 26638 ts_tablet_manager.cc:1905] T 93c9e4253d5844b5a89569e2035537ba P a25c0921c520499aa06c828f01791c1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:28.196651 26638 ts_tablet_manager.cc:1918] T 93c9e4253d5844b5a89569e2035537ba P a25c0921c520499aa06c828f01791c1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:40:28.196712 26638 log.cc:1196] T 93c9e4253d5844b5a89569e2035537ba P a25c0921c520499aa06c828f01791c1d: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-2/wal/wals/93c9e4253d5844b5a89569e2035537ba
I20260928 06:40:28.196961 26638 ts_tablet_manager.cc:1939] T 93c9e4253d5844b5a89569e2035537ba P a25c0921c520499aa06c828f01791c1d: Deleting consensus metadata
I20260928 06:40:28.197360 26639 tablet_replica.cc:330] T 93c9e4253d5844b5a89569e2035537ba P becf12ef2f814e66b83a1ddee4fa138b: stopping tablet replica
I20260928 06:40:28.197471 26639 raft_consensus.cc:2229] T 93c9e4253d5844b5a89569e2035537ba P becf12ef2f814e66b83a1ddee4fa138b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:40:28.197585 26639 raft_consensus.cc:2258] T 93c9e4253d5844b5a89569e2035537ba P becf12ef2f814e66b83a1ddee4fa138b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 06:40:28.197731 26474 connection.cc:535] server connection from 127.0.0.1:43736 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 06:40:28.197786 26203 connection.cc:535] server connection from 127.0.0.1:38940 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 06:40:28.198235 26639 ts_tablet_manager.cc:1905] T 93c9e4253d5844b5a89569e2035537ba P becf12ef2f814e66b83a1ddee4fa138b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:28.199817 26639 ts_tablet_manager.cc:1918] T 93c9e4253d5844b5a89569e2035537ba P becf12ef2f814e66b83a1ddee4fa138b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:40:28.199891 26639 log.cc:1196] T 93c9e4253d5844b5a89569e2035537ba P becf12ef2f814e66b83a1ddee4fa138b: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-1/wal/wals/93c9e4253d5844b5a89569e2035537ba
I20260928 06:40:28.200119 26639 ts_tablet_manager.cc:1939] T 93c9e4253d5844b5a89569e2035537ba P becf12ef2f814e66b83a1ddee4fa138b: Deleting consensus metadata
W20260928 06:40:28.200491 26337 connection.cc:441] server connection from 127.0.0.1:49618 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49618 (request call id 1) could send its response
I20260928 06:40:28.200968 26599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:28.191902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58684 (local address 127.18.211.124:40615)
0928 06:40:28.192230 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:28.192232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:28.192242 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:40:28.192264 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:28.192265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:28.192280 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:28.192333 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:28.193341 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:28.194026 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:28.195403 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:28.195703 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:28.195748 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:28.197387 (+ 1639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:28.197389 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:28.197389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:28.197396 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:28.199265 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:28.199736 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:28.199737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:28.199738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:28.199797 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:28.200001 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:28.200002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:28.200003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:28.200181 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:28.200292 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:28.200769 (+ 477us) server_negotiation.cc:299] Negotiation successful
0928 06:40:28.200867 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302}
I20260928 06:40:28.206655 25999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:28.201123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56794 (local address 127.18.211.125:37143)
0928 06:40:28.201167 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:28.201169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:28.201176 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:40:28.201195 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:28.201196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:28.201205 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:28.201248 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:28.201777 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:28.202236 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:28.202894 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:28.203131 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:28.203193 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:28.204207 (+ 1014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:28.204208 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:28.204208 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:28.204215 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:28.205513 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:28.205885 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:28.205886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:28.205886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:28.205944 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:28.206135 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:28.206137 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:28.206137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:28.206268 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:28.206370 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:28.206488 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 06:40:28.206556 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:40:28.208451 26319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:28.200541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58692 (local address 127.18.211.124:40615)
0928 06:40:28.200577 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:28.200578 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:28.201514 (+ 936us) server_negotiation.cc:407] Connection header received
0928 06:40:28.201617 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:28.201618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:28.201628 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:28.201671 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:28.202427 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:28.203174 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:28.204173 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:28.204458 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:28.204498 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:28.205944 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:28.205945 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:28.205946 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:28.205952 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:28.207420 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:28.207824 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:28.207825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:28.207825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:28.207870 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:28.208053 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:28.208053 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:28.208053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:28.208138 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:28.208202 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:28.208333 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 06:40:28.208390 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:40:29.187489 26647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.187228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44085 (local address 127.18.211.66:51081)
0928 06:40:29.187366 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:29.187437 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
I20260928 06:40:29.187583 26648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.187326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44085 (local address 127.18.211.67:47765)
0928 06:40:29.187460 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:29.187534 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
W20260928 06:40:29.187794 26587 heartbeater.cc:644] Failed to heartbeat to 127.18.211.126:44085 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.211.126:44085: connect: Connection refused (error 111)
W20260928 06:40:29.187917 26451 heartbeater.cc:644] Failed to heartbeat to 127.18.211.126:44085 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.211.126:44085: connect: Connection refused (error 111)
I20260928 06:40:29.188194 26649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.187916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44085 (local address 127.18.211.65:34299)
0928 06:40:29.188052 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:29.188122 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":60,"threads_started":1}
W20260928 06:40:29.188480 26314 heartbeater.cc:644] Failed to heartbeat to 127.18.211.126:44085 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.211.126:44085: connect: Connection refused (error 111)
I20260928 06:40:29.690728 26650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f5d90cb26f1a4900a3e9a927bb0d79be)
I20260928 06:40:29.690881 26650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: false } } peers { permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 40615 } attrs { promote: false } }
I20260928 06:40:29.691174 26650 leader_election.cc:288] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f5d90cb26f1a4900a3e9a927bb0d79be (127.18.211.126:44085), 1295d210e86447f6ba916d9371e50885 (127.18.211.125:37143)
I20260928 06:40:29.691257 26651 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f5d90cb26f1a4900a3e9a927bb0d79be)
I20260928 06:40:29.691465 26651 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: false } } peers { permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 40615 } attrs { promote: false } }
I20260928 06:40:29.691548 26652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.691325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44085 (local address 127.0.0.1:35858)
0928 06:40:29.691462 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:29.691510 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":62,"threads_started":1}
W20260928 06:40:29.691648 26118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.211.126:44085: connect: Connection refused (error 111)
I20260928 06:40:29.691726 26651 leader_election.cc:288] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f5d90cb26f1a4900a3e9a927bb0d79be (127.18.211.126:44085), 90b91ba24e8c4c20853a01477d44c493 (127.18.211.124:40615)
I20260928 06:40:29.692034 26656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.691798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44085 (local address 127.0.0.1:35864)
0928 06:40:29.691938 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:29.691997 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":43,"threads_started":1}
I20260928 06:40:29.692061 26652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.691983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44085 (local address 127.0.0.1:35870)
0928 06:40:29.692019 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:29.692039 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 06:40:29.692131 26118 leader_election.cc:334] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f5d90cb26f1a4900a3e9a927bb0d79be (127.18.211.126:44085): Network error: Client connection negotiation failed: client connection to 127.18.211.126:44085: connect: Connection refused (error 111)
W20260928 06:40:29.692320 25932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.211.126:44085: connect: Connection refused (error 111)
I20260928 06:40:29.692840 26659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.692604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44085 (local address 127.0.0.1:35884)
0928 06:40:29.692772 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:29.692807 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":73,"threads_started":1}
W20260928 06:40:29.692919 25932 leader_election.cc:334] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f5d90cb26f1a4900a3e9a927bb0d79be (127.18.211.126:44085): Network error: Client connection negotiation failed: client connection to 127.18.211.126:44085: connect: Connection refused (error 111)
I20260928 06:40:29.694271 26654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.691418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55390 (local address 127.18.211.125:37143)
0928 06:40:29.691514 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:29.691516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:29.691524 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:40:29.691547 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:29.691547 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:29.691557 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:29.691617 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:29.691810 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:29.692533 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:29.693585 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:29.693967 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:29.694029 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:29.694039 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:29.694158 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 06:40:29.694194 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":79,"thread_start_us":43,"threads_started":1}
I20260928 06:40:29.694271 26653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.691309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.125:37143 (local address 127.0.0.1:55390)
0928 06:40:29.691423 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:29.691435 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:29.691490 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:29.691668 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:29.691668 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:29.691671 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:29.691735 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:29.691737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:29.692611 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:29.692612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:29.693501 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:29.693504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:29.694049 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:29.694050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:29.694100 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:29.694104 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:40:29.694148 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 06:40:29.694194 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":45,"threads_started":1}
I20260928 06:40:29.694501 25958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "90b91ba24e8c4c20853a01477d44c493" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1295d210e86447f6ba916d9371e50885" is_pre_election: true
I20260928 06:40:29.694598 25958 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90b91ba24e8c4c20853a01477d44c493 in term 1.
I20260928 06:40:29.694746 26120 leader_election.cc:302] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1295d210e86447f6ba916d9371e50885, 90b91ba24e8c4c20853a01477d44c493; no voters: f5d90cb26f1a4900a3e9a927bb0d79be
I20260928 06:40:29.694860 26650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 06:40:29.694916 26650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f5d90cb26f1a4900a3e9a927bb0d79be)
I20260928 06:40:29.694936 26650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:40:29.694994 26657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.692044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54252 (local address 127.18.211.124:40615)
0928 06:40:29.692179 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:29.692181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:29.692350 (+ 169us) server_negotiation.cc:407] Connection header received
0928 06:40:29.692414 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:29.692415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:29.692424 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:29.692484 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:29.692632 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:29.693228 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:29.694137 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:29.694670 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:29.694753 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:29.694761 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:29.694903 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 06:40:29.694943 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
I20260928 06:40:29.695003 26656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.692239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:40615 (local address 127.0.0.1:54252)
0928 06:40:29.692284 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:29.692293 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:29.692349 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:29.692493 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:29.692494 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:29.692496 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:29.692553 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:29.692555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:29.693302 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:29.693303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:29.694061 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:29.694063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:29.694778 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:29.694779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:29.694841 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:29.694845 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:40:29.694897 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 06:40:29.694934 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 06:40:29.695214 26143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1295d210e86447f6ba916d9371e50885" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "90b91ba24e8c4c20853a01477d44c493" is_pre_election: true
I20260928 06:40:29.695744 26650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: false } } peers { permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 40615 } attrs { promote: false } }
I20260928 06:40:29.695902 26650 leader_election.cc:288] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [CANDIDATE]: Term 2 election: Requested vote from peers f5d90cb26f1a4900a3e9a927bb0d79be (127.18.211.126:44085), 1295d210e86447f6ba916d9371e50885 (127.18.211.125:37143)
I20260928 06:40:29.695912 26143 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1295d210e86447f6ba916d9371e50885 in current term 2: Already voted for candidate 90b91ba24e8c4c20853a01477d44c493 in this term.
I20260928 06:40:29.696115 26653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.695969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44085 (local address 127.0.0.1:35894)
0928 06:40:29.696063 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:29.696093 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 06:40:29.696091 25958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "90b91ba24e8c4c20853a01477d44c493" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1295d210e86447f6ba916d9371e50885"
I20260928 06:40:29.696138 25932 leader_election.cc:302] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1295d210e86447f6ba916d9371e50885; no voters: 90b91ba24e8c4c20853a01477d44c493, f5d90cb26f1a4900a3e9a927bb0d79be
I20260928 06:40:29.696163 25958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:40:29.696482 26653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.696392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44085 (local address 127.0.0.1:35904)
0928 06:40:29.696434 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:29.696459 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260928 06:40:29.696551 26118 leader_election.cc:334] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f5d90cb26f1a4900a3e9a927bb0d79be (127.18.211.126:44085): Network error: Client connection negotiation failed: client connection to 127.18.211.126:44085: connect: Connection refused (error 111)
I20260928 06:40:29.696966 25958 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90b91ba24e8c4c20853a01477d44c493 in term 2.
I20260928 06:40:29.697038 26651 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:40:29.697124 26120 leader_election.cc:302] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1295d210e86447f6ba916d9371e50885, 90b91ba24e8c4c20853a01477d44c493; no voters: f5d90cb26f1a4900a3e9a927bb0d79be
I20260928 06:40:29.697247 26650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:40:29.697394 26650 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [term 2 LEADER]: Becoming Leader. State: Replica: 90b91ba24e8c4c20853a01477d44c493, State: Running, Role: LEADER
I20260928 06:40:29.697456 26650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: false } } peers { permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 40615 } attrs { promote: false } }
I20260928 06:40:29.697764 26660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90b91ba24e8c4c20853a01477d44c493. Latest consensus state: current_term: 2 leader_uuid: "90b91ba24e8c4c20853a01477d44c493" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: false } } peers { permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 40615 } attrs { promote: false } } }
I20260928 06:40:29.697853 26660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:29.698040 26662 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:40:29.698400 26662 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:40:29.698469 26662 catalog_manager.cc:1259] Loaded cluster ID: dce0b6ad27964fd8a9550617d17fad5b
I20260928 06:40:29.698542 26662 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:40:29.698684 26662 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:40:29.698737 26662 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493: Loaded TSK: 0
I20260928 06:40:29.698827 26662 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:40:29.744252 26133 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:40:29.745913 26133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:40:29.746922 26653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.746768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44085 (local address 127.0.0.1:35908)
0928 06:40:29.746848 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:29.746888 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 06:40:29.746906 25958 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [term 2 FOLLOWER]: Refusing update from remote peer 90b91ba24e8c4c20853a01477d44c493: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 06:40:29.747270 26653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.747165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44085 (local address 127.0.0.1:35924)
0928 06:40:29.747212 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:29.747244 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 06:40:29.747327 26650 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260928 06:40:29.747467 26118 consensus_peers.cc:487] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 -> Peer f5d90cb26f1a4900a3e9a927bb0d79be (127.18.211.126:44085): Couldn't send request to peer f5d90cb26f1a4900a3e9a927bb0d79be. Status: Network error: Client connection negotiation failed: client connection to 127.18.211.126:44085: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 06:40:29.747979 26651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90b91ba24e8c4c20853a01477d44c493. Latest consensus state: current_term: 2 leader_uuid: "90b91ba24e8c4c20853a01477d44c493" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: false } } peers { permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 40615 } attrs { promote: false } } }
I20260928 06:40:29.748068 26651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:40:29.748588 26650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "90b91ba24e8c4c20853a01477d44c493" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: false } } peers { permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 40615 } attrs { promote: false } } }
I20260928 06:40:29.748709 26650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493 [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:29.748800 26651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "90b91ba24e8c4c20853a01477d44c493" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f5d90cb26f1a4900a3e9a927bb0d79be" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44085 } } peers { permanent_uuid: "1295d210e86447f6ba916d9371e50885" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37143 } attrs { promote: false } } peers { permanent_uuid: "90b91ba24e8c4c20853a01477d44c493" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 40615 } attrs { promote: false } } }
I20260928 06:40:29.748955 26651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1295d210e86447f6ba916d9371e50885 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:40:29.753383 26652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.750183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:44887 (local address 127.0.0.1:44230)
0928 06:40:29.750296 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:29.750307 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:29.750351 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:29.750542 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:29.750543 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:29.750546 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:29.750600 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:29.750602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:29.751403 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:29.751404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:29.752420 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:29.752423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:29.753153 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:29.753154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:29.753223 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:29.753227 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:40:29.753276 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 06:40:29.753319 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260928 06:40:29.753476 26653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.750254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:35203 (local address 127.0.0.1:58214)
0928 06:40:29.750380 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:29.750390 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:29.750429 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:29.750766 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:29.750767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:29.750769 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:29.750809 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:29.750811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:29.751736 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:29.751737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:29.752512 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:29.752515 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:29.753233 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:29.753234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:29.753298 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:29.753302 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:40:29.753350 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 06:40:29.753391 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":38}
I20260928 06:40:29.753472 26671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.750314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58214 (local address 127.18.211.65:35203)
0928 06:40:29.750616 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:29.750618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:29.750625 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:40:29.750648 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:29.750649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:29.750658 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:29.750719 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:29.750884 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:29.751650 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:29.752604 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:29.753157 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:29.753233 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:29.753244 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:29.753362 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 06:40:29.753400 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":51,"threads_started":1}
I20260928 06:40:29.753679 26247 tablet_service.cc:1467] Processing CreateTablet for tablet 1f70d53f316a472ba1576301a5e0a978 (DEFAULT_TABLE table=test-table [id=4f5b738a51f04fa684f2f0fba350bb70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:29.753748 26669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.750255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44230 (local address 127.18.211.66:44887)
0928 06:40:29.750429 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:29.750431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:29.750438 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:40:29.750463 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:29.750464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:29.750475 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:29.750540 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:29.750713 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:29.751333 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:29.752510 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:29.753077 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:29.753150 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:29.753163 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:29.753631 (+ 468us) server_negotiation.cc:299] Negotiation successful
0928 06:40:29.753679 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":66,"threads_started":1}
I20260928 06:40:29.753813 26247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f70d53f316a472ba1576301a5e0a978. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:29.753894 26670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.750263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:39055 (local address 127.0.0.1:59494)
0928 06:40:29.750511 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:29.750520 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:29.750568 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:29.750792 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:29.750793 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:29.750795 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:29.750824 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:29.750825 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:29.751723 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:29.751724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:29.752819 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:29.752822 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:29.753673 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:29.753675 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:29.753735 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:29.753739 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:40:29.753783 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 06:40:29.753821 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":26,"thread_start_us":82,"threads_started":1}
I20260928 06:40:29.754034 26384 tablet_service.cc:1467] Processing CreateTablet for tablet 1f70d53f316a472ba1576301a5e0a978 (DEFAULT_TABLE table=test-table [id=4f5b738a51f04fa684f2f0fba350bb70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:29.754110 26672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.750410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59494 (local address 127.18.211.67:39055)
0928 06:40:29.750530 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:29.750532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:29.750559 (+ 27us) server_negotiation.cc:407] Connection header received
0928 06:40:29.750650 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:29.750651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:29.750661 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:29.750741 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:29.750893 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:29.751644 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:29.752889 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:29.753349 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:29.753405 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:29.753417 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:29.754016 (+ 599us) server_negotiation.cc:299] Negotiation successful
0928 06:40:29.754061 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":52,"threads_started":1}
I20260928 06:40:29.754156 26384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f70d53f316a472ba1576301a5e0a978. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:29.754343 26520 tablet_service.cc:1467] Processing CreateTablet for tablet 1f70d53f316a472ba1576301a5e0a978 (DEFAULT_TABLE table=test-table [id=4f5b738a51f04fa684f2f0fba350bb70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:29.754431 26520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f70d53f316a472ba1576301a5e0a978. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:29.755568 26674 tablet_bootstrap.cc:492] T 1f70d53f316a472ba1576301a5e0a978 P becf12ef2f814e66b83a1ddee4fa138b: Bootstrap starting.
I20260928 06:40:29.755668 26675 tablet_bootstrap.cc:492] T 1f70d53f316a472ba1576301a5e0a978 P a25c0921c520499aa06c828f01791c1d: Bootstrap starting.
I20260928 06:40:29.755811 26673 tablet_bootstrap.cc:492] T 1f70d53f316a472ba1576301a5e0a978 P 538426449b59422daa1419fc97acf5a6: Bootstrap starting.
I20260928 06:40:29.755920 26674 tablet_bootstrap.cc:654] T 1f70d53f316a472ba1576301a5e0a978 P becf12ef2f814e66b83a1ddee4fa138b: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:29.756035 26675 tablet_bootstrap.cc:654] T 1f70d53f316a472ba1576301a5e0a978 P a25c0921c520499aa06c828f01791c1d: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:29.756176 26673 tablet_bootstrap.cc:654] T 1f70d53f316a472ba1576301a5e0a978 P 538426449b59422daa1419fc97acf5a6: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:29.756556 26674 tablet_bootstrap.cc:492] T 1f70d53f316a472ba1576301a5e0a978 P becf12ef2f814e66b83a1ddee4fa138b: No bootstrap required, opened a new log
I20260928 06:40:29.756603 26674 ts_tablet_manager.cc:1397] T 1f70d53f316a472ba1576301a5e0a978 P becf12ef2f814e66b83a1ddee4fa138b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:29.756724 26674 raft_consensus.cc:348] T 1f70d53f316a472ba1576301a5e0a978 P becf12ef2f814e66b83a1ddee4fa138b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "becf12ef2f814e66b83a1ddee4fa138b" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 44887 } } peers { permanent_uuid: "538426449b59422daa1419fc97acf5a6" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35203 } } peers { permanent_uuid: "a25c0921c520499aa06c828f01791c1d" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 39055 } }
I20260928 06:40:29.756780 26674 raft_consensus.cc:374] T 1f70d53f316a472ba1576301a5e0a978 P becf12ef2f814e66b83a1ddee4fa138b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:29.756798 26674 raft_consensus.cc:729] T 1f70d53f316a472ba1576301a5e0a978 P becf12ef2f814e66b83a1ddee4fa138b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: becf12ef2f814e66b83a1ddee4fa138b, State: Initialized, Role: FOLLOWER
I20260928 06:40:29.756841 26674 consensus_queue.cc:260] T 1f70d53f316a472ba1576301a5e0a978 P becf12ef2f814e66b83a1ddee4fa138b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "becf12ef2f814e66b83a1ddee4fa138b" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 44887 } } peers { permanent_uuid: "538426449b59422daa1419fc97acf5a6" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35203 } } peers { permanent_uuid: "a25c0921c520499aa06c828f01791c1d" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 39055 } }
I20260928 06:40:29.756971 26674 ts_tablet_manager.cc:1428] T 1f70d53f316a472ba1576301a5e0a978 P becf12ef2f814e66b83a1ddee4fa138b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:40:29.757328 26675 tablet_bootstrap.cc:492] T 1f70d53f316a472ba1576301a5e0a978 P a25c0921c520499aa06c828f01791c1d: No bootstrap required, opened a new log
I20260928 06:40:29.757328 26673 tablet_bootstrap.cc:492] T 1f70d53f316a472ba1576301a5e0a978 P 538426449b59422daa1419fc97acf5a6: No bootstrap required, opened a new log
I20260928 06:40:29.757377 26675 ts_tablet_manager.cc:1397] T 1f70d53f316a472ba1576301a5e0a978 P a25c0921c520499aa06c828f01791c1d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:40:29.757377 26673 ts_tablet_manager.cc:1397] T 1f70d53f316a472ba1576301a5e0a978 P 538426449b59422daa1419fc97acf5a6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:40:29.757483 26675 raft_consensus.cc:348] T 1f70d53f316a472ba1576301a5e0a978 P a25c0921c520499aa06c828f01791c1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "becf12ef2f814e66b83a1ddee4fa138b" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 44887 } } peers { permanent_uuid: "538426449b59422daa1419fc97acf5a6" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35203 } } peers { permanent_uuid: "a25c0921c520499aa06c828f01791c1d" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 39055 } }
I20260928 06:40:29.757540 26675 raft_consensus.cc:374] T 1f70d53f316a472ba1576301a5e0a978 P a25c0921c520499aa06c828f01791c1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:29.757513 26673 raft_consensus.cc:348] T 1f70d53f316a472ba1576301a5e0a978 P 538426449b59422daa1419fc97acf5a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "becf12ef2f814e66b83a1ddee4fa138b" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 44887 } } peers { permanent_uuid: "538426449b59422daa1419fc97acf5a6" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35203 } } peers { permanent_uuid: "a25c0921c520499aa06c828f01791c1d" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 39055 } }
I20260928 06:40:29.757557 26675 raft_consensus.cc:729] T 1f70d53f316a472ba1576301a5e0a978 P a25c0921c520499aa06c828f01791c1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a25c0921c520499aa06c828f01791c1d, State: Initialized, Role: FOLLOWER
I20260928 06:40:29.757568 26673 raft_consensus.cc:374] T 1f70d53f316a472ba1576301a5e0a978 P 538426449b59422daa1419fc97acf5a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:29.757586 26673 raft_consensus.cc:729] T 1f70d53f316a472ba1576301a5e0a978 P 538426449b59422daa1419fc97acf5a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 538426449b59422daa1419fc97acf5a6, State: Initialized, Role: FOLLOWER
I20260928 06:40:29.757597 26675 consensus_queue.cc:260] T 1f70d53f316a472ba1576301a5e0a978 P a25c0921c520499aa06c828f01791c1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "becf12ef2f814e66b83a1ddee4fa138b" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 44887 } } peers { permanent_uuid: "538426449b59422daa1419fc97acf5a6" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35203 } } peers { permanent_uuid: "a25c0921c520499aa06c828f01791c1d" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 39055 } }
I20260928 06:40:29.757628 26673 consensus_queue.cc:260] T 1f70d53f316a472ba1576301a5e0a978 P 538426449b59422daa1419fc97acf5a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "becf12ef2f814e66b83a1ddee4fa138b" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 44887 } } peers { permanent_uuid: "538426449b59422daa1419fc97acf5a6" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35203 } } peers { permanent_uuid: "a25c0921c520499aa06c828f01791c1d" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 39055 } }
I20260928 06:40:29.757738 26675 ts_tablet_manager.cc:1428] T 1f70d53f316a472ba1576301a5e0a978 P a25c0921c520499aa06c828f01791c1d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:29.757758 26673 ts_tablet_manager.cc:1428] T 1f70d53f316a472ba1576301a5e0a978 P 538426449b59422daa1419fc97acf5a6: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:40:29.758368 26450 heartbeater.cc:503] Master 127.18.211.124:40615 was elected leader, sending a full tablet report...
I20260928 06:40:29.758421 26586 heartbeater.cc:503] Master 127.18.211.124:40615 was elected leader, sending a full tablet report...
I20260928 06:40:29.758127 26680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.757118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44085 (local address 127.18.211.66:49893)
0928 06:40:29.758053 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:29.758094 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":911,"thread_start_us":79,"threads_started":1}
I20260928 06:40:29.758922 26684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.758367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44085 (local address 127.18.211.65:45223)
0928 06:40:29.758623 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:29.758874 (+ 251us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":76,"threads_started":1}
I20260928 06:40:29.758952 26683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.757918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44085 (local address 127.18.211.67:35393)
0928 06:40:29.758734 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:29.758774 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":80,"threads_started":1}
I20260928 06:40:29.759044 26680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.758965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44085 (local address 127.18.211.66:38839)
0928 06:40:29.759000 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:29.759022 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 06:40:29.759125 26313 heartbeater.cc:503] Master 127.18.211.124:40615 was elected leader, sending a full tablet report...
I20260928 06:40:29.759347 26685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.759157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44085 (local address 127.18.211.67:34077)
0928 06:40:29.759278 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:29.759315 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":74,"threads_started":1}
I20260928 06:40:29.759539 26684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.759419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44085 (local address 127.18.211.65:47861)
0928 06:40:29.759454 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:29.759486 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260928 06:40:29.834995 26679 raft_consensus.cc:482] T 1f70d53f316a472ba1576301a5e0a978 P becf12ef2f814e66b83a1ddee4fa138b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:40:29.835125 26679 raft_consensus.cc:504] T 1f70d53f316a472ba1576301a5e0a978 P becf12ef2f814e66b83a1ddee4fa138b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "becf12ef2f814e66b83a1ddee4fa138b" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 44887 } } peers { permanent_uuid: "538426449b59422daa1419fc97acf5a6" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35203 } } peers { permanent_uuid: "a25c0921c520499aa06c828f01791c1d" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 39055 } }
I20260928 06:40:29.835496 26679 leader_election.cc:288] T 1f70d53f316a472ba1576301a5e0a978 P becf12ef2f814e66b83a1ddee4fa138b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 538426449b59422daa1419fc97acf5a6 (127.18.211.65:35203), a25c0921c520499aa06c828f01791c1d (127.18.211.67:39055)
I20260928 06:40:29.839570 26672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.835751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:44299 (local address 127.18.211.67:39055)
0928 06:40:29.835800 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:29.835802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:29.835872 (+ 70us) server_negotiation.cc:407] Connection header received
0928 06:40:29.835954 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:29.835955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:29.835964 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:29.836026 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:29.836192 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:29.836991 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:29.838580 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:29.839191 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:29.839277 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:29.839285 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:29.839454 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 06:40:29.839495 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:40:29.839570 26671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.835798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:54859 (local address 127.18.211.65:35203)
0928 06:40:29.835855 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:29.835857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:29.835944 (+ 87us) server_negotiation.cc:407] Connection header received
0928 06:40:29.835997 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:29.835999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:29.836007 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:29.836070 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:29.836255 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:29.836990 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:29.838580 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:29.839200 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:29.839272 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:29.839283 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:29.839454 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 06:40:29.839495 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 06:40:29.839586 26680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.835674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:39055 (local address 127.18.211.66:44299)
0928 06:40:29.835806 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:29.835817 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:29.835884 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:29.836051 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:29.836052 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:29.836054 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:29.836132 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:29.836134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:29.837091 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:29.837092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:29.838478 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:29.838480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:29.839313 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:29.839314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:29.839377 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:29.839381 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:40:29.839441 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 06:40:29.839493 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":90}
I20260928 06:40:29.839596 26686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.835692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:35203 (local address 127.18.211.66:54859)
0928 06:40:29.835882 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:29.835891 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:29.835943 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:29.836081 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:29.836082 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:29.836083 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:29.836137 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:29.836138 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:29.837091 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:29.837092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:29.838472 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:29.838477 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:29.839290 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:29.839291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:29.839377 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:29.839382 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:40:29.839441 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 06:40:29.839493 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":14,"thread_start_us":111,"threads_started":1}
I20260928 06:40:29.839879 26540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f70d53f316a472ba1576301a5e0a978" candidate_uuid: "becf12ef2f814e66b83a1ddee4fa138b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a25c0921c520499aa06c828f01791c1d" is_pre_election: true
I20260928 06:40:29.840020 26540 raft_consensus.cc:2454] T 1f70d53f316a472ba1576301a5e0a978 P a25c0921c520499aa06c828f01791c1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate becf12ef2f814e66b83a1ddee4fa138b in term 0.
I20260928 06:40:29.840255 26338 leader_election.cc:302] T 1f70d53f316a472ba1576301a5e0a978 P becf12ef2f814e66b83a1ddee4fa138b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a25c0921c520499aa06c828f01791c1d, becf12ef2f814e66b83a1ddee4fa138b; no voters:
I20260928 06:40:29.840289 26267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f70d53f316a472ba1576301a5e0a978" candidate_uuid: "becf12ef2f814e66b83a1ddee4fa138b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "538426449b59422daa1419fc97acf5a6" is_pre_election: true
I20260928 06:40:29.840378 26267 raft_consensus.cc:2454] T 1f70d53f316a472ba1576301a5e0a978 P 538426449b59422daa1419fc97acf5a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate becf12ef2f814e66b83a1ddee4fa138b in term 0.
I20260928 06:40:29.840462 26679 raft_consensus.cc:2781] T 1f70d53f316a472ba1576301a5e0a978 P becf12ef2f814e66b83a1ddee4fa138b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:40:29.840505 26679 raft_consensus.cc:482] T 1f70d53f316a472ba1576301a5e0a978 P becf12ef2f814e66b83a1ddee4fa138b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:40:29.840524 26679 raft_consensus.cc:3037] T 1f70d53f316a472ba1576301a5e0a978 P becf12ef2f814e66b83a1ddee4fa138b [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:29.841320 26679 raft_consensus.cc:504] T 1f70d53f316a472ba1576301a5e0a978 P becf12ef2f814e66b83a1ddee4fa138b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "becf12ef2f814e66b83a1ddee4fa138b" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 44887 } } peers { permanent_uuid: "538426449b59422daa1419fc97acf5a6" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35203 } } peers { permanent_uuid: "a25c0921c520499aa06c828f01791c1d" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 39055 } }
I20260928 06:40:29.841444 26679 leader_election.cc:288] T 1f70d53f316a472ba1576301a5e0a978 P becf12ef2f814e66b83a1ddee4fa138b [CANDIDATE]: Term 1 election: Requested vote from peers 538426449b59422daa1419fc97acf5a6 (127.18.211.65:35203), a25c0921c520499aa06c828f01791c1d (127.18.211.67:39055)
I20260928 06:40:29.841600 26267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f70d53f316a472ba1576301a5e0a978" candidate_uuid: "becf12ef2f814e66b83a1ddee4fa138b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "538426449b59422daa1419fc97acf5a6"
I20260928 06:40:29.841668 26267 raft_consensus.cc:3037] T 1f70d53f316a472ba1576301a5e0a978 P 538426449b59422daa1419fc97acf5a6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:29.842298 26267 raft_consensus.cc:2454] T 1f70d53f316a472ba1576301a5e0a978 P 538426449b59422daa1419fc97acf5a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate becf12ef2f814e66b83a1ddee4fa138b in term 1.
I20260928 06:40:29.842413 26540 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f70d53f316a472ba1576301a5e0a978" candidate_uuid: "becf12ef2f814e66b83a1ddee4fa138b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a25c0921c520499aa06c828f01791c1d"
I20260928 06:40:29.842470 26337 leader_election.cc:302] T 1f70d53f316a472ba1576301a5e0a978 P becf12ef2f814e66b83a1ddee4fa138b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 538426449b59422daa1419fc97acf5a6, becf12ef2f814e66b83a1ddee4fa138b; no voters:
I20260928 06:40:29.842491 26540 raft_consensus.cc:3037] T 1f70d53f316a472ba1576301a5e0a978 P a25c0921c520499aa06c828f01791c1d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:29.842552 26679 raft_consensus.cc:2781] T 1f70d53f316a472ba1576301a5e0a978 P becf12ef2f814e66b83a1ddee4fa138b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:29.842720 26679 raft_consensus.cc:686] T 1f70d53f316a472ba1576301a5e0a978 P becf12ef2f814e66b83a1ddee4fa138b [term 1 LEADER]: Becoming Leader. State: Replica: becf12ef2f814e66b83a1ddee4fa138b, State: Running, Role: LEADER
I20260928 06:40:29.842849 26679 consensus_queue.cc:237] T 1f70d53f316a472ba1576301a5e0a978 P becf12ef2f814e66b83a1ddee4fa138b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "becf12ef2f814e66b83a1ddee4fa138b" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 44887 } } peers { permanent_uuid: "538426449b59422daa1419fc97acf5a6" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35203 } } peers { permanent_uuid: "a25c0921c520499aa06c828f01791c1d" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 39055 } }
I20260928 06:40:29.843315 26540 raft_consensus.cc:2454] T 1f70d53f316a472ba1576301a5e0a978 P a25c0921c520499aa06c828f01791c1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate becf12ef2f814e66b83a1ddee4fa138b in term 1.
I20260928 06:40:29.843621 26680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.843530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44085 (local address 127.18.211.66:42611)
0928 06:40:29.843562 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:29.843591 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
I20260928 06:40:29.843776 26134 catalog_manager.cc:5495] T 1f70d53f316a472ba1576301a5e0a978 P becf12ef2f814e66b83a1ddee4fa138b reported cstate change: term changed from 0 to 1, leader changed from <none> to becf12ef2f814e66b83a1ddee4fa138b (127.18.211.66). New cstate: current_term: 1 leader_uuid: "becf12ef2f814e66b83a1ddee4fa138b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "becf12ef2f814e66b83a1ddee4fa138b" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 44887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "538426449b59422daa1419fc97acf5a6" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a25c0921c520499aa06c828f01791c1d" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 39055 } health_report { overall_health: UNKNOWN } } }
I20260928 06:40:29.870121 26669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.867874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44236 (local address 127.18.211.66:44887)
0928 06:40:29.867938 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:29.867940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:29.867949 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:40:29.868003 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:29.868003 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:29.868025 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:29.868083 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:40:29.868207 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:29.868682 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:29.869421 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:29.869609 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:29.869666 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:29.869815 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:40:29.869914 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:40:29.869976 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:29.870022 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 06:40:29.870045 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:40:29.871209 26267 raft_consensus.cc:1261] T 1f70d53f316a472ba1576301a5e0a978 P 538426449b59422daa1419fc97acf5a6 [term 1 FOLLOWER]: Refusing update from remote peer becf12ef2f814e66b83a1ddee4fa138b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:40:29.871320 26540 raft_consensus.cc:1261] T 1f70d53f316a472ba1576301a5e0a978 P a25c0921c520499aa06c828f01791c1d [term 1 FOLLOWER]: Refusing update from remote peer becf12ef2f814e66b83a1ddee4fa138b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:40:29.871493 26687 consensus_queue.cc:1035] T 1f70d53f316a472ba1576301a5e0a978 P becf12ef2f814e66b83a1ddee4fa138b [LEADER]: Connected to new peer: Peer: permanent_uuid: "538426449b59422daa1419fc97acf5a6" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 35203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:40:29.871609 26679 consensus_queue.cc:1035] T 1f70d53f316a472ba1576301a5e0a978 P becf12ef2f814e66b83a1ddee4fa138b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a25c0921c520499aa06c828f01791c1d" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 39055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:40:29.872643 26684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.872544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44085 (local address 127.18.211.65:35213)
0928 06:40:29.872582 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:29.872612 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 06:40:29.872953 26685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.872735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44085 (local address 127.18.211.67:38107)
0928 06:40:29.872893 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:29.872922 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260928 06:40:29.873821 26692 mvcc.cc:205] Tried to move back new op lower bound from 7334205971950374912 to 7334205971836936192. Current Snapshot: MvccSnapshot[applied={T|T < 7334205971950374912}]
I20260928 06:40:29.873900 26680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.873738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44085 (local address 127.18.211.66:38141)
0928 06:40:29.873841 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:29.873870 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260928 06:40:29.874475 26685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.874393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44085 (local address 127.18.211.67:46769)
0928 06:40:29.874427 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:29.874450 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260928 06:40:29.875348 26680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.875205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44085 (local address 127.18.211.66:34941)
0928 06:40:29.875297 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:29.875323 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 06:40:29.875643 26691 mvcc.cc:205] Tried to move back new op lower bound from 7334205971950374912 to 7334205971836936192. Current Snapshot: MvccSnapshot[applied={T|T < 7334205971950374912}]
I20260928 06:40:29.875737 26684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.875557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44085 (local address 127.18.211.65:52089)
0928 06:40:29.875675 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:29.875704 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:44085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260928 06:40:29.876081 26134 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:40:29.876164 26134 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:40:29.877247 26134 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 90b91ba24e8c4c20853a01477d44c493: Sending DeleteTablet for 3 replicas of tablet 1f70d53f316a472ba1576301a5e0a978
I20260928 06:40:29.877579 26520 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f70d53f316a472ba1576301a5e0a978 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:29 UTC) from {username='oryx', principal='oryx/127.18.211.124@KRBTEST.COM'} at 127.0.0.1:59494
I20260928 06:40:29.877579 26384 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f70d53f316a472ba1576301a5e0a978 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:29 UTC) from {username='oryx', principal='oryx/127.18.211.124@KRBTEST.COM'} at 127.0.0.1:44230
I20260928 06:40:29.877871 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 26096
I20260928 06:40:29.877908 26247 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f70d53f316a472ba1576301a5e0a978 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:29 UTC) from {username='oryx', principal='oryx/127.18.211.124@KRBTEST.COM'} at 127.0.0.1:58214
I20260928 06:40:29.879225 26700 tablet_replica.cc:330] T 1f70d53f316a472ba1576301a5e0a978 P becf12ef2f814e66b83a1ddee4fa138b: stopping tablet replica
I20260928 06:40:29.879276 26700 raft_consensus.cc:2229] T 1f70d53f316a472ba1576301a5e0a978 P becf12ef2f814e66b83a1ddee4fa138b [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:40:29.879348 26700 raft_consensus.cc:2258] T 1f70d53f316a472ba1576301a5e0a978 P becf12ef2f814e66b83a1ddee4fa138b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:29.879442 26701 tablet_replica.cc:330] T 1f70d53f316a472ba1576301a5e0a978 P a25c0921c520499aa06c828f01791c1d: stopping tablet replica
I20260928 06:40:29.879482 26701 raft_consensus.cc:2229] T 1f70d53f316a472ba1576301a5e0a978 P a25c0921c520499aa06c828f01791c1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:40:29.879522 26701 raft_consensus.cc:2258] T 1f70d53f316a472ba1576301a5e0a978 P a25c0921c520499aa06c828f01791c1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:29.879842 26701 ts_tablet_manager.cc:1905] T 1f70d53f316a472ba1576301a5e0a978 P a25c0921c520499aa06c828f01791c1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:29.880151 26700 ts_tablet_manager.cc:1905] T 1f70d53f316a472ba1576301a5e0a978 P becf12ef2f814e66b83a1ddee4fa138b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:29.881376 26701 ts_tablet_manager.cc:1918] T 1f70d53f316a472ba1576301a5e0a978 P a25c0921c520499aa06c828f01791c1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:40:29.881492 26701 log.cc:1196] T 1f70d53f316a472ba1576301a5e0a978 P a25c0921c520499aa06c828f01791c1d: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-2/wal/wals/1f70d53f316a472ba1576301a5e0a978
I20260928 06:40:29.881819 26701 ts_tablet_manager.cc:1939] T 1f70d53f316a472ba1576301a5e0a978 P a25c0921c520499aa06c828f01791c1d: Deleting consensus metadata
I20260928 06:40:29.881915 26700 ts_tablet_manager.cc:1918] T 1f70d53f316a472ba1576301a5e0a978 P becf12ef2f814e66b83a1ddee4fa138b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:40:29.882048 26700 log.cc:1196] T 1f70d53f316a472ba1576301a5e0a978 P becf12ef2f814e66b83a1ddee4fa138b: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790577602522428-19277-0/minicluster-data/ts-1/wal/wals/1f70d53f316a472ba1576301a5e0a978
I20260928 06:40:29.882170 26119 catalog_manager.cc:4841] TS a25c0921c520499aa06c828f01791c1d (127.18.211.67:39055): tablet 1f70d53f316a472ba1576301a5e0a978 (table test-table [id=4f5b738a51f04fa684f2f0fba350bb70]) successfully deleted
I20260928 06:40:29.882354 26700 ts_tablet_manager.cc:1939] T 1f70d53f316a472ba1576301a5e0a978 P becf12ef2f814e66b83a1ddee4fa138b: Deleting consensus metadata
I20260928 06:40:29.882647 26119 catalog_manager.cc:4841] TS becf12ef2f814e66b83a1ddee4fa138b (127.18.211.66:44887): tablet 1f70d53f316a472ba1576301a5e0a978 (table test-table [id=4f5b738a51f04fa684f2f0fba350bb70]) successfully deleted
I20260928 06:40:29.883703 26670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:29.883580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:35203 (local address 127.0.0.1:58216)
0928 06:40:29.883638 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:29.883671 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.65:35203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 06:40:29.883850 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 26321
W20260928 06:40:29.883863 26118 catalog_manager.cc:4584] TS 538426449b59422daa1419fc97acf5a6 (127.18.211.65:35203): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1f70d53f316a472ba1576301a5e0a978: Network error: Client connection negotiation failed: client connection to 127.18.211.65:35203: connect: Connection refused (error 111)
I20260928 06:40:29.888868 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 26457
I20260928 06:40:29.895750 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 25815
I20260928 06:40:29.900132 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 26002
2026-09-28T06:40:29Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4705 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:40:29 dist-test-slave-b28r krb5kdc[26704](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:40:29 dist-test-slave-b28r krb5kdc[26704](info): set up 2 sockets
Sep 28 06:40:29 dist-test-slave-b28r krb5kdc[26704](info): commencing operation
krb5kdc: starting...
W20260928 06:40:30.848296 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:40:30 dist-test-slave-b28r krb5kdc[26704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:40:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:40:30Z Disabled control of system clock
WARNING: no policy specified for oryx/127.18.211.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.211.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.211.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.126.keytab.
Entry for principal oryx/127.18.211.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.126.keytab.
I20260928 06:40:30.955880 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:41149
--webserver_interface=127.18.211.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.126.keytab
--principal=oryx/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:42543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:41149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:31.035962 26717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:31.036181 26717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:31.036206 26717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:31.037322 26717 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:40:31.037361 26717 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 06:40:31.037376 26717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:31.037389 26717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:31.037402 26717 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:40:31.037415 26717 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:40:31.038897 26717 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:42543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:41149
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.126.keytab
--principal=oryx/127.18.211.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:41149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:31.039130 26717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:31.039336 26717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:31.043926 26722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:31.043936 26723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:31.043937 26725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:31.044155 26717 server_base.cc:1034] running on GCE node
I20260928 06:40:31.044371 26717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:31.044644 26717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:31.045779 26717 hybrid_clock.cc:648] HybridClock initialized: now 1790577631045764 us; error 34 us; skew 500 ppm
Sep 28 06:40:31 dist-test-slave-b28r krb5kdc[26704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577631, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:31.053421 26717 init.cc:377] Logged in from keytab as oryx/127.18.211.126@KRBTEST.COM (short username oryx)
I20260928 06:40:31.053891 26717 webserver.cc:460] Webserver started at http://127.18.211.126:45565/ using document root <none> and password file <none>
I20260928 06:40:31.054106 26717 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:31.054154 26717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:31.054279 26717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:31.055369 26717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "29696cea0aef44818a5a20643af487e8"
format_stamp: "Formatted at 2026-09-28 06:40:31 on dist-test-slave-b28r"
server_key: "fd8cf28ba44a43771e456c1cbc3d5904"
server_key_iv: "8f7eff1f5aa548c3223f384cc7b5d00c"
server_key_version: "encryptionkey@0"
I20260928 06:40:31.055697 26717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "29696cea0aef44818a5a20643af487e8"
format_stamp: "Formatted at 2026-09-28 06:40:31 on dist-test-slave-b28r"
server_key: "fd8cf28ba44a43771e456c1cbc3d5904"
server_key_iv: "8f7eff1f5aa548c3223f384cc7b5d00c"
server_key_version: "encryptionkey@0"
I20260928 06:40:31.056900 26717 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:40:31.057606 26732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:31.057762 26717 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:40:31.057857 26717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "29696cea0aef44818a5a20643af487e8"
format_stamp: "Formatted at 2026-09-28 06:40:31 on dist-test-slave-b28r"
server_key: "fd8cf28ba44a43771e456c1cbc3d5904"
server_key_iv: "8f7eff1f5aa548c3223f384cc7b5d00c"
server_key_version: "encryptionkey@0"
I20260928 06:40:31.057914 26717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:31.101444 26717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:31.103483 26717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:31.103631 26717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:31.108009 26717 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:41149
I20260928 06:40:31.108094 26794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:41149 every 8 connection(s)
I20260928 06:40:31.108390 26717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:40:31.109082 26795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:31.110716 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 26717
I20260928 06:40:31.110832 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:40:31.111032 19277 external_mini_cluster.cc:1428] Setting key d7a6d8a18e60695d346f46369617732e
I20260928 06:40:31.111769 26795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29696cea0aef44818a5a20643af487e8: Bootstrap starting.
I20260928 06:40:31.112367 26795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29696cea0aef44818a5a20643af487e8: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:31.112658 26795 log.cc:824] T 00000000000000000000000000000000 P 29696cea0aef44818a5a20643af487e8: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:31.113392 26795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29696cea0aef44818a5a20643af487e8: No bootstrap required, opened a new log
I20260928 06:40:31.115227 26795 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29696cea0aef44818a5a20643af487e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29696cea0aef44818a5a20643af487e8" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41149 } }
I20260928 06:40:31.115382 26795 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29696cea0aef44818a5a20643af487e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:31.115417 26795 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29696cea0aef44818a5a20643af487e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29696cea0aef44818a5a20643af487e8, State: Initialized, Role: FOLLOWER
I20260928 06:40:31.115592 26795 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29696cea0aef44818a5a20643af487e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29696cea0aef44818a5a20643af487e8" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41149 } }
I20260928 06:40:31.115685 26795 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29696cea0aef44818a5a20643af487e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:40:31.115716 26795 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29696cea0aef44818a5a20643af487e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:40:31.115769 26795 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29696cea0aef44818a5a20643af487e8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:31.116417 26795 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29696cea0aef44818a5a20643af487e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29696cea0aef44818a5a20643af487e8" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41149 } }
I20260928 06:40:31.116532 26795 leader_election.cc:302] T 00000000000000000000000000000000 P 29696cea0aef44818a5a20643af487e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29696cea0aef44818a5a20643af487e8; no voters:
I20260928 06:40:31.116704 26795 leader_election.cc:288] T 00000000000000000000000000000000 P 29696cea0aef44818a5a20643af487e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:40:31.116750 26800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29696cea0aef44818a5a20643af487e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:31.116923 26800 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29696cea0aef44818a5a20643af487e8 [term 1 LEADER]: Becoming Leader. State: Replica: 29696cea0aef44818a5a20643af487e8, State: Running, Role: LEADER
I20260928 06:40:31.116981 26795 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29696cea0aef44818a5a20643af487e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:40:31.117116 26800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29696cea0aef44818a5a20643af487e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29696cea0aef44818a5a20643af487e8" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41149 } }
Sep 28 06:40:31 dist-test-slave-b28r krb5kdc[26704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM
I20260928 06:40:31.117570 26802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29696cea0aef44818a5a20643af487e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29696cea0aef44818a5a20643af487e8. Latest consensus state: current_term: 1 leader_uuid: "29696cea0aef44818a5a20643af487e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29696cea0aef44818a5a20643af487e8" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41149 } } }
I20260928 06:40:31.117658 26802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29696cea0aef44818a5a20643af487e8 [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:31.117791 26801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29696cea0aef44818a5a20643af487e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29696cea0aef44818a5a20643af487e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29696cea0aef44818a5a20643af487e8" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41149 } } }
I20260928 06:40:31.117871 26801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29696cea0aef44818a5a20643af487e8 [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:31.117995 26807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:40:31.118580 26807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:40:31.120399 26807 catalog_manager.cc:1347] Generated new cluster ID: 7b1df94ccbbb4791acdeba27fef47231
I20260928 06:40:31.120453 26807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:40:31.121665 26798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:31.111616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36258 (local address 127.18.211.126:41149)
0928 06:40:31.111819 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:31.111830 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:31.111868 (+ 38us) server_negotiation.cc:407] Connection header received
0928 06:40:31.112334 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:31.112342 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:31.112391 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:31.112539 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:31.113274 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.114195 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.114940 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.115392 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.115497 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:31.118174 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:31.118178 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:31.118178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:31.118190 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:31.120160 (+ 1970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:31.120572 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:31.120575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:31.120575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:31.120671 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:31.120863 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:31.120864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:31.120864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:31.121100 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:31.121249 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:31.121358 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 06:40:31.121494 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
I20260928 06:40:31.141044 26807 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:40:31.141588 26807 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:40:31.145603 26807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29696cea0aef44818a5a20643af487e8: Generated new TSK 0
I20260928 06:40:31.145799 26807 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.18.211.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.211.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.211.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.65.keytab.
Entry for principal oryx/127.18.211.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.65.keytab.
I20260928 06:40:31.177661 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.65:0
--local_ip_for_outbound_sockets=127.18.211.65
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:41149
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.65.keytab
--principal=oryx/127.18.211.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:42543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:31.263430 26821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:31.263633 26821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:31.263655 26821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:31.264792 26821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:31.264833 26821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:31.264851 26821 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:40:31.264870 26821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.65
I20260928 06:40:31.266350 26821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:42543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.65.keytab
--principal=oryx/127.18.211.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.211.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.211.126:41149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.65
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:31.266543 26821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:31.266723 26821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:31.271616 26827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:31.271644 26826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:31.271629 26829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:31.271817 26821 server_base.cc:1034] running on GCE node
I20260928 06:40:31.272011 26821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:31.272239 26821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:31.273416 26821 hybrid_clock.cc:648] HybridClock initialized: now 1790577631273399 us; error 32 us; skew 500 ppm
Sep 28 06:40:31 dist-test-slave-b28r krb5kdc[26704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577631, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:31.276037 26821 init.cc:377] Logged in from keytab as oryx/127.18.211.65@KRBTEST.COM (short username oryx)
I20260928 06:40:31.276507 26821 webserver.cc:460] Webserver started at http://127.18.211.65:41275/ using document root <none> and password file <none>
I20260928 06:40:31.276710 26821 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:31.276750 26821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:31.276852 26821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:31.277894 26821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-0/data/instance:
uuid: "f034fbd6323b41a28e36c21af4b7f475"
format_stamp: "Formatted at 2026-09-28 06:40:31 on dist-test-slave-b28r"
server_key: "bc8d5fb300b032412cf5e15b4abe8eee"
server_key_iv: "b95c0f89802bd3829acaaf650f2de77b"
server_key_version: "encryptionkey@0"
I20260928 06:40:31.278196 26821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance:
uuid: "f034fbd6323b41a28e36c21af4b7f475"
format_stamp: "Formatted at 2026-09-28 06:40:31 on dist-test-slave-b28r"
server_key: "bc8d5fb300b032412cf5e15b4abe8eee"
server_key_iv: "b95c0f89802bd3829acaaf650f2de77b"
server_key_version: "encryptionkey@0"
I20260928 06:40:31.279450 26821 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260928 06:40:31.280333 26836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:31.280561 26821 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:31.280653 26821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-0/wal
uuid: "f034fbd6323b41a28e36c21af4b7f475"
format_stamp: "Formatted at 2026-09-28 06:40:31 on dist-test-slave-b28r"
server_key: "bc8d5fb300b032412cf5e15b4abe8eee"
server_key_iv: "b95c0f89802bd3829acaaf650f2de77b"
server_key_version: "encryptionkey@0"
I20260928 06:40:31.280720 26821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:31.310401 26821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:31.312348 26821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:31.312492 26821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:31.313015 26821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:31.313068 26821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:31.313102 26821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:31.313118 26821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 06:40:31 dist-test-slave-b28r krb5kdc[26704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577631, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM
I20260928 06:40:31.320144 26821 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.65:45105
I20260928 06:40:31.320303 26951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.65:45105 every 8 connection(s)
I20260928 06:40:31.320557 26821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
I20260928 06:40:31.322491 26798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:31.313543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:42597 (local address 127.18.211.126:41149)
0928 06:40:31.313609 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:31.313612 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:31.313863 (+ 251us) server_negotiation.cc:407] Connection header received
0928 06:40:31.314550 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:31.314552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:31.314561 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:31.314603 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:31.315481 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.316062 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.317124 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.317441 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.317518 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:31.319749 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:31.319750 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:31.319751 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:31.319764 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:31.321212 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:31.321588 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:31.321589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:31.321590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:31.321660 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:31.321889 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:31.321889 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:31.321890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:31.322003 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:31.322058 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:31.322315 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 06:40:31.322402 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:40:31.322634 26845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:31.313510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:41149 (local address 127.18.211.65:42597)
0928 06:40:31.313675 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:31.313725 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:31.314423 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:31.314696 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:31.314701 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:31.315062 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:31.315380 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:31.315388 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.316150 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.316151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:31.317045 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:31.317048 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.317536 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.317537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:31.317701 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:31.318209 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:31.318218 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:31.319662 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:31.321289 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:31.321295 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:31.321297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:31.321522 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:31.321714 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:31.321714 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:31.321722 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:31.321833 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:31.322056 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:31.322062 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:31.322196 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 06:40:31.322282 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 06:40:31.322432 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260928 06:40:31.323061 26952 heartbeater.cc:347] Connected to a master server at 127.18.211.126:41149
I20260928 06:40:31.323171 26952 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:31.323359 26952 heartbeater.cc:511] Master 127.18.211.126:41149 requested a full tablet report, sending...
I20260928 06:40:31.323426 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 26821
I20260928 06:40:31.323530 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance
I20260928 06:40:31.323745 19277 external_mini_cluster.cc:1428] Setting key 96a775992a9a186b06dfcb716094a4c4
I20260928 06:40:31.323961 26748 ts_manager.cc:194] Registered new tserver with Master: f034fbd6323b41a28e36c21af4b7f475 (127.18.211.65:45105)
I20260928 06:40:31.324798 26748 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.211.65@KRBTEST.COM'} at 127.18.211.65:42597
I20260928 06:40:31.330547 26958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:31.324091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:41149 (local address 127.18.211.65:59715)
0928 06:40:31.324207 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:31.324218 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:31.324278 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:31.325337 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:31.325338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:31.325486 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:31.325533 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:31.325534 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.326398 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.326399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:31.326914 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:31.326916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.327374 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.327375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:31.327431 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:31.327830 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:31.327834 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:31.328110 (+ 276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:31.329680 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:31.329681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:31.329681 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:31.329851 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:31.330035 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:31.330035 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:31.330035 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:31.330082 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:31.330319 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:31.330320 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:31.330358 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 06:40:31.330403 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 06:40:31.330475 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":59,"threads_started":1}
I20260928 06:40:31.330601 26798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:31.324994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:59715 (local address 127.18.211.126:41149)
0928 06:40:31.325188 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:31.325190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:31.325196 (+ 6us) server_negotiation.cc:407] Connection header received
0928 06:40:31.325211 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:31.325212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:31.325220 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:31.325267 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:31.325598 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.326287 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.326993 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.327280 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.327347 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:31.328192 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:31.328194 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:31.328194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:31.328201 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:31.329597 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:31.329919 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:31.329920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:31.329920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:31.329979 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:31.330134 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:31.330135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:31.330135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:31.330258 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:31.330317 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:31.330446 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 06:40:31.330514 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
WARNING: no policy specified for oryx/127.18.211.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.211.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.211.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.66.keytab.
Entry for principal oryx/127.18.211.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.66.keytab.
I20260928 06:40:31.347491 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.66:0
--local_ip_for_outbound_sockets=127.18.211.66
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:41149
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.66.keytab
--principal=oryx/127.18.211.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:42543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:31.426386 26961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:31.426612 26961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:31.426648 26961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:31.427903 26961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:31.427950 26961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:31.427978 26961 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:40:31.428000 26961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.66
I20260928 06:40:31.429643 26961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:42543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.66.keytab
--principal=oryx/127.18.211.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.211.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.211.126:41149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.66
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:31.429881 26961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:31.430090 26961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:31.434796 26969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:31.434880 26967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:31.434836 26966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:31.434998 26961 server_base.cc:1034] running on GCE node
I20260928 06:40:31.435266 26961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:31.435467 26961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:31.436595 26961 hybrid_clock.cc:648] HybridClock initialized: now 1790577631436572 us; error 38 us; skew 500 ppm
Sep 28 06:40:31 dist-test-slave-b28r krb5kdc[26704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577631, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:31.439030 26961 init.cc:377] Logged in from keytab as oryx/127.18.211.66@KRBTEST.COM (short username oryx)
I20260928 06:40:31.439489 26961 webserver.cc:460] Webserver started at http://127.18.211.66:43149/ using document root <none> and password file <none>
I20260928 06:40:31.439702 26961 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:31.439744 26961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:31.439865 26961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:31.440869 26961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-1/data/instance:
uuid: "0007b185a0014c199414c8c0607806f1"
format_stamp: "Formatted at 2026-09-28 06:40:31 on dist-test-slave-b28r"
server_key: "adbc81760cc5dd5eb47d0f5f1c190e75"
server_key_iv: "4bb2e92639a12bc60a97de149e9a2926"
server_key_version: "encryptionkey@0"
I20260928 06:40:31.441341 26961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance:
uuid: "0007b185a0014c199414c8c0607806f1"
format_stamp: "Formatted at 2026-09-28 06:40:31 on dist-test-slave-b28r"
server_key: "adbc81760cc5dd5eb47d0f5f1c190e75"
server_key_iv: "4bb2e92639a12bc60a97de149e9a2926"
server_key_version: "encryptionkey@0"
I20260928 06:40:31.442616 26961 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:40:31.443368 26976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:31.443555 26961 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:31.443645 26961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-1/wal
uuid: "0007b185a0014c199414c8c0607806f1"
format_stamp: "Formatted at 2026-09-28 06:40:31 on dist-test-slave-b28r"
server_key: "adbc81760cc5dd5eb47d0f5f1c190e75"
server_key_iv: "4bb2e92639a12bc60a97de149e9a2926"
server_key_version: "encryptionkey@0"
I20260928 06:40:31.443724 26961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:31.457787 26961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:31.459580 26961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:31.459694 26961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:31.460213 26961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:31.460254 26961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:31.460282 26961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:31.460299 26961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:31.466566 26961 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.66:36675
I20260928 06:40:31.466658 27091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.66:36675 every 8 connection(s)
Sep 28 06:40:31 dist-test-slave-b28r krb5kdc[26704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577631, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM
I20260928 06:40:31.466995 26961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
I20260928 06:40:31.469595 26798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:31.460651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:44787 (local address 127.18.211.126:41149)
0928 06:40:31.460696 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:31.460698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:31.460958 (+ 260us) server_negotiation.cc:407] Connection header received
0928 06:40:31.461421 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:31.461422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:31.461434 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:31.461506 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:31.462290 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.463046 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.464536 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.464836 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.464914 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:31.467326 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:31.467328 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:31.467328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:31.467336 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:31.468514 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:31.468845 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:31.468846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:31.468846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:31.468894 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:31.469061 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:31.469061 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:31.469061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:31.469152 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:31.469198 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:31.469465 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 06:40:31.469533 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:40:31.469676 26985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:31.460665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:41149 (local address 127.18.211.66:44787)
0928 06:40:31.460827 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:31.460860 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:31.461330 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:31.461579 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:31.461583 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:31.461915 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:31.462185 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:31.462191 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.463198 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.463199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:31.464450 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:31.464454 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.464913 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.464914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:31.465015 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:31.465503 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:31.465514 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:31.467242 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:31.468579 (+ 1337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:31.468584 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:31.468586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:31.468790 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:31.468936 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:31.468937 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:31.468937 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:31.469022 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:31.469191 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:31.469194 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:31.469323 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 06:40:31.469402 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 06:40:31.469514 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":90,"threads_started":1}
I20260928 06:40:31.470091 27092 heartbeater.cc:347] Connected to a master server at 127.18.211.126:41149
I20260928 06:40:31.470171 27092 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:31.470350 27092 heartbeater.cc:511] Master 127.18.211.126:41149 requested a full tablet report, sending...
I20260928 06:40:31.470767 26749 ts_manager.cc:194] Registered new tserver with Master: 0007b185a0014c199414c8c0607806f1 (127.18.211.66:36675)
I20260928 06:40:31.471261 26749 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.211.66@KRBTEST.COM'} at 127.18.211.66:44787
I20260928 06:40:31.471932 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 26961
I20260928 06:40:31.472039 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance
I20260928 06:40:31.472220 19277 external_mini_cluster.cc:1428] Setting key 8796ab5c26eff7749e5725753633245f
I20260928 06:40:31.476522 26798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:31.471075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:60383 (local address 127.18.211.126:41149)
0928 06:40:31.471128 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:31.471130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:31.471275 (+ 145us) server_negotiation.cc:407] Connection header received
0928 06:40:31.471301 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:31.471301 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:31.471312 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:31.471367 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:31.471669 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.472319 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.473032 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.473319 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.473383 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:31.474151 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:31.474153 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:31.474153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:31.474159 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:31.475388 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:31.475710 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:31.475710 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:31.475711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:31.475756 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:31.475953 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:31.475955 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:31.475955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:31.476154 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:31.476259 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:31.476398 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 06:40:31.476462 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 06:40:31.476562 27098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:31.471021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:41149 (local address 127.18.211.66:60383)
0928 06:40:31.471181 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:31.471190 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:31.471268 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:31.471370 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:31.471370 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:31.471536 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:31.471593 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:31.471595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.472416 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.472417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:31.472965 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:31.472967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.473395 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.473396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:31.473444 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:31.473823 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:31.473827 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:31.474072 (+ 245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:31.475458 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:31.475459 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:31.475459 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:31.475648 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:31.475809 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:31.475809 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:31.475809 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:31.475873 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:31.476252 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:31.476253 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:31.476300 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 06:40:31.476366 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 06:40:31.476464 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":84,"threads_started":1}
WARNING: no policy specified for oryx/127.18.211.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.211.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.211.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.67.keytab.
Entry for principal oryx/127.18.211.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.67.keytab.
I20260928 06:40:31.495066 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.67:0
--local_ip_for_outbound_sockets=127.18.211.67
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:41149
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.67.keytab
--principal=oryx/127.18.211.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:42543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:31.574069 27101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:31.574272 27101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:31.574296 27101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:31.575470 27101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:31.575515 27101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:31.575533 27101 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 06:40:31.575549 27101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.67
I20260928 06:40:31.577026 27101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:42543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.67.keytab
--principal=oryx/127.18.211.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.211.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.211.126:41149
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.67
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:31.577232 27101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:31.577450 27101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:31.582170 27106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:31.582165 27107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:31.582216 27109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:31.582610 27101 server_base.cc:1034] running on GCE node
I20260928 06:40:31.582798 27101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:31.583029 27101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:31.584198 27101 hybrid_clock.cc:648] HybridClock initialized: now 1790577631584179 us; error 33 us; skew 500 ppm
Sep 28 06:40:31 dist-test-slave-b28r krb5kdc[26704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577631, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:31.586558 27101 init.cc:377] Logged in from keytab as oryx/127.18.211.67@KRBTEST.COM (short username oryx)
I20260928 06:40:31.586982 27101 webserver.cc:460] Webserver started at http://127.18.211.67:34285/ using document root <none> and password file <none>
I20260928 06:40:31.587215 27101 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:31.587271 27101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:31.587361 27101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:31.588373 27101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-2/data/instance:
uuid: "17f7bd3caba1497985a5d062a28ec8b9"
format_stamp: "Formatted at 2026-09-28 06:40:31 on dist-test-slave-b28r"
server_key: "f33a0d9ebc430b9835a141af9bcff975"
server_key_iv: "82e644146c022b6247b2f7b73006624d"
server_key_version: "encryptionkey@0"
I20260928 06:40:31.588703 27101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance:
uuid: "17f7bd3caba1497985a5d062a28ec8b9"
format_stamp: "Formatted at 2026-09-28 06:40:31 on dist-test-slave-b28r"
server_key: "f33a0d9ebc430b9835a141af9bcff975"
server_key_iv: "82e644146c022b6247b2f7b73006624d"
server_key_version: "encryptionkey@0"
I20260928 06:40:31.589936 27101 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:40:31.590562 27116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:31.590734 27101 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:40:31.590806 27101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-2/wal
uuid: "17f7bd3caba1497985a5d062a28ec8b9"
format_stamp: "Formatted at 2026-09-28 06:40:31 on dist-test-slave-b28r"
server_key: "f33a0d9ebc430b9835a141af9bcff975"
server_key_iv: "82e644146c022b6247b2f7b73006624d"
server_key_version: "encryptionkey@0"
I20260928 06:40:31.590858 27101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:31.604178 27101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:31.606050 27101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:31.606166 27101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:31.606647 27101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:31.606689 27101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:31.606715 27101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:31.606729 27101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:31.612869 27101 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.67:35041
I20260928 06:40:31.612972 27231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.67:35041 every 8 connection(s)
I20260928 06:40:31.613312 27101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:40:31 dist-test-slave-b28r krb5kdc[26704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577631, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM
I20260928 06:40:31.616219 26798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:31.607169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:52041 (local address 127.18.211.126:41149)
0928 06:40:31.607217 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:31.607220 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:31.607447 (+ 227us) server_negotiation.cc:407] Connection header received
0928 06:40:31.608000 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:31.608002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:31.608016 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:31.608081 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:31.608904 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.609656 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.611154 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.611444 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.611522 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:31.613890 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:31.613891 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:31.613892 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:31.613899 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:31.615214 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:31.615560 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:31.615561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:31.615562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:31.615609 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:31.615764 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:31.615765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:31.615765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:31.615860 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:31.615906 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:31.616102 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 06:40:31.616158 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:40:31.616394 27125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:31.607175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:41149 (local address 127.18.211.67:52041)
0928 06:40:31.607335 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:31.607371 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:31.607865 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:31.608173 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:31.608177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:31.608504 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:31.608806 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:31.608812 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.609759 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.609759 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:31.611026 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:31.611029 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.611523 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.611524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:31.611641 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:31.612239 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:31.612250 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:31.613808 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:31.615293 (+ 1485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:31.615300 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:31.615302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:31.615503 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:31.615645 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:31.615645 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:31.615645 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:31.615727 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:31.615903 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:31.615906 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:31.616015 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 06:40:31.616091 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 06:40:31.616203 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":76,"threads_started":1}
I20260928 06:40:31.616871 27232 heartbeater.cc:347] Connected to a master server at 127.18.211.126:41149
I20260928 06:40:31.616948 27232 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:31.617115 27232 heartbeater.cc:511] Master 127.18.211.126:41149 requested a full tablet report, sending...
I20260928 06:40:31.617533 26748 ts_manager.cc:194] Registered new tserver with Master: 17f7bd3caba1497985a5d062a28ec8b9 (127.18.211.67:35041)
I20260928 06:40:31.618009 26748 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.211.67@KRBTEST.COM'} at 127.18.211.67:52041
I20260928 06:40:31.619343 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 27101
I20260928 06:40:31.619470 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance
I20260928 06:40:31.619675 19277 external_mini_cluster.cc:1428] Setting key d91027b4966921b21f8b6b85b1e5d35f
I20260928 06:40:31.620450 19277 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:40:31.623660 26798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:31.617972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:36531 (local address 127.18.211.126:41149)
0928 06:40:31.618013 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:31.618016 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:31.618024 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:40:31.618078 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:31.618078 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:31.618087 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:31.618135 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:31.618458 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.619045 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.619972 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.620244 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.620303 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:31.621193 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:31.621194 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:31.621194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:31.621200 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:31.622567 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:31.622931 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:31.622932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:31.622932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:31.622990 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:31.623187 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:31.623188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:31.623188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:31.623309 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:31.623367 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:31.623503 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 06:40:31.623569 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:40:31.623667 27238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:31.617851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:41149 (local address 127.18.211.67:36531)
0928 06:40:31.617964 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:31.617974 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:31.618022 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:31.618147 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:31.618149 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:31.618319 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:31.618388 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:31.618390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.619138 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.619139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:31.619882 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:31.619885 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.620312 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.620312 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:31.620375 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:31.620733 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:31.620738 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:31.621118 (+ 380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:31.622648 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:31.622649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:31.622650 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:31.622852 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:31.623042 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:31.623042 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:31.623042 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:31.623120 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:31.623366 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:31.623367 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:31.623411 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 06:40:31.623471 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 06:40:31.623571 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":54,"threads_started":1}
Sep 28 06:40:31 dist-test-slave-b28r krb5kdc[26704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577631, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 06:40:31 dist-test-slave-b28r krb5kdc[26704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:31 dist-test-slave-b28r krb5kdc[26704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 06:40:31.641525 26798 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:31.636564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36274 (local address 127.18.211.126:41149)
0928 06:40:31.636617 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:31.636619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:31.636627 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:40:31.636651 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:31.636652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:31.636660 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:31.636716 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:31.637277 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.637769 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.638453 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.638637 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.638705 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:31.641444 (+ 2739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36274: BlockingRecv error: recv got EOF from 127.0.0.1:36274 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 28 06:40:31 dist-test-slave-b28r krb5kdc[26704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577631, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM
I20260928 06:40:31.648622 26798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:31.642245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36286 (local address 127.18.211.126:41149)
0928 06:40:31.642290 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:31.642292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:31.642364 (+ 72us) server_negotiation.cc:407] Connection header received
0928 06:40:31.642413 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:31.642414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:31.642421 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:31.642467 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:31.642852 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.643349 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.644006 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.644175 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.644220 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:31.646342 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:31.646344 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:31.646344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:31.646350 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:31.647647 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:31.647974 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:31.647975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:31.647975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:31.648020 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:31.648216 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:31.648216 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:31.648217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:31.648334 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:31.648404 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:31.648503 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 06:40:31.648556 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:40:31.650174 26748 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:40:31.651826 26748 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:40:31.658207 27254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:31.654084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:36675 (local address 127.0.0.1:60706)
0928 06:40:31.654473 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:31.654481 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:31.654516 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:31.654765 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:31.654765 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:31.654767 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:31.654825 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:31.654826 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.656192 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.656193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:31.657271 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:31.657273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.657900 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.657901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:31.657979 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:31.657993 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 06:40:31.658057 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 06:40:31.658118 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":5,"thread_start_us":92,"threads_started":1}
I20260928 06:40:31.658262 27258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:31.654365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58852 (local address 127.18.211.67:35041)
0928 06:40:31.654560 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:31.654563 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:31.654571 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:40:31.654603 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:31.654605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:31.654619 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:31.654899 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:31.655027 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.656138 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.657187 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.657901 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.658002 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:31.658067 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:31.658134 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 06:40:31.658169 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":77,"threads_started":1}
I20260928 06:40:31.658488 27257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:31.654342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60706 (local address 127.18.211.66:36675)
0928 06:40:31.654521 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:31.654527 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:31.654538 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:40:31.654586 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:31.654596 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:31.654613 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:31.654697 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:31.654931 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.656101 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.657340 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.657825 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.657890 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:31.657965 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:31.658377 (+ 412us) server_negotiation.cc:299] Negotiation successful
0928 06:40:31.658413 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":76,"threads_started":1}
I20260928 06:40:31.658517 27259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:31.654547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45992 (local address 127.18.211.65:45105)
0928 06:40:31.654725 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:31.654730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:31.654767 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:40:31.654836 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:31.654838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:31.654853 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:31.654921 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:31.655170 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.655852 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.657234 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.657919 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.657975 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:31.658050 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:31.658390 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 06:40:31.658431 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":87,"threads_started":1}
I20260928 06:40:31.658206 27255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:31.654083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:35041 (local address 127.0.0.1:58852)
0928 06:40:31.654262 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:31.654307 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:31.654378 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:31.654746 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:31.654750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:31.654752 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:31.654824 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:31.654826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.656244 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.656245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:31.657116 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:31.657119 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.658008 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.658009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:31.658061 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:31.658064 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:40:31.658117 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 06:40:31.658146 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":61,"threads_started":1}
I20260928 06:40:31.658330 27256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:31.654083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:45105 (local address 127.0.0.1:45992)
0928 06:40:31.654716 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:31.654725 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:31.654777 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:31.654984 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:31.654985 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:31.654987 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:31.655044 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:31.655046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.655932 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.655933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:31.657166 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:31.657169 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.658086 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.658087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:31.658158 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:31.658162 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:40:31.658209 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 06:40:31.658251 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":3,"thread_start_us":115,"threads_started":1}
I20260928 06:40:31.659299 27026 tablet_service.cc:1467] Processing CreateTablet for tablet 4b44d9339a83464eb45ddad7182dc9fa (DEFAULT_TABLE table=test-table [id=f1db164cdcdf4a66be4e2d659e0a55c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:31.659483 26886 tablet_service.cc:1467] Processing CreateTablet for tablet 4b44d9339a83464eb45ddad7182dc9fa (DEFAULT_TABLE table=test-table [id=f1db164cdcdf4a66be4e2d659e0a55c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:31.659512 27166 tablet_service.cc:1467] Processing CreateTablet for tablet 4b44d9339a83464eb45ddad7182dc9fa (DEFAULT_TABLE table=test-table [id=f1db164cdcdf4a66be4e2d659e0a55c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:31.659670 27026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b44d9339a83464eb45ddad7182dc9fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:31.659760 26886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b44d9339a83464eb45ddad7182dc9fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:31.659801 27166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b44d9339a83464eb45ddad7182dc9fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:31.662683 27260 tablet_bootstrap.cc:492] T 4b44d9339a83464eb45ddad7182dc9fa P 17f7bd3caba1497985a5d062a28ec8b9: Bootstrap starting.
I20260928 06:40:31.663241 27260 tablet_bootstrap.cc:654] T 4b44d9339a83464eb45ddad7182dc9fa P 17f7bd3caba1497985a5d062a28ec8b9: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:31.663395 27262 tablet_bootstrap.cc:492] T 4b44d9339a83464eb45ddad7182dc9fa P f034fbd6323b41a28e36c21af4b7f475: Bootstrap starting.
I20260928 06:40:31.663561 27260 log.cc:824] T 4b44d9339a83464eb45ddad7182dc9fa P 17f7bd3caba1497985a5d062a28ec8b9: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:31.663944 27262 tablet_bootstrap.cc:654] T 4b44d9339a83464eb45ddad7182dc9fa P f034fbd6323b41a28e36c21af4b7f475: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:31.664239 27262 log.cc:824] T 4b44d9339a83464eb45ddad7182dc9fa P f034fbd6323b41a28e36c21af4b7f475: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:31.664296 27260 tablet_bootstrap.cc:492] T 4b44d9339a83464eb45ddad7182dc9fa P 17f7bd3caba1497985a5d062a28ec8b9: No bootstrap required, opened a new log
I20260928 06:40:31.664357 27260 ts_tablet_manager.cc:1397] T 4b44d9339a83464eb45ddad7182dc9fa P 17f7bd3caba1497985a5d062a28ec8b9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:40:31.664552 27261 tablet_bootstrap.cc:492] T 4b44d9339a83464eb45ddad7182dc9fa P 0007b185a0014c199414c8c0607806f1: Bootstrap starting.
I20260928 06:40:31.665120 27261 tablet_bootstrap.cc:654] T 4b44d9339a83464eb45ddad7182dc9fa P 0007b185a0014c199414c8c0607806f1: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:31.665423 27261 log.cc:824] T 4b44d9339a83464eb45ddad7182dc9fa P 0007b185a0014c199414c8c0607806f1: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:31.665959 27262 tablet_bootstrap.cc:492] T 4b44d9339a83464eb45ddad7182dc9fa P f034fbd6323b41a28e36c21af4b7f475: No bootstrap required, opened a new log
I20260928 06:40:31.666029 27262 ts_tablet_manager.cc:1397] T 4b44d9339a83464eb45ddad7182dc9fa P f034fbd6323b41a28e36c21af4b7f475: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 06:40:31.666087 27261 tablet_bootstrap.cc:492] T 4b44d9339a83464eb45ddad7182dc9fa P 0007b185a0014c199414c8c0607806f1: No bootstrap required, opened a new log
I20260928 06:40:31.666152 27261 ts_tablet_manager.cc:1397] T 4b44d9339a83464eb45ddad7182dc9fa P 0007b185a0014c199414c8c0607806f1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:40:31.666626 27260 raft_consensus.cc:348] T 4b44d9339a83464eb45ddad7182dc9fa P 17f7bd3caba1497985a5d062a28ec8b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } } peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } }
I20260928 06:40:31.666795 27260 raft_consensus.cc:374] T 4b44d9339a83464eb45ddad7182dc9fa P 17f7bd3caba1497985a5d062a28ec8b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:31.666831 27260 raft_consensus.cc:729] T 4b44d9339a83464eb45ddad7182dc9fa P 17f7bd3caba1497985a5d062a28ec8b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17f7bd3caba1497985a5d062a28ec8b9, State: Initialized, Role: FOLLOWER
I20260928 06:40:31.666985 27260 consensus_queue.cc:260] T 4b44d9339a83464eb45ddad7182dc9fa P 17f7bd3caba1497985a5d062a28ec8b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } } peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } }
I20260928 06:40:31.667266 27260 ts_tablet_manager.cc:1428] T 4b44d9339a83464eb45ddad7182dc9fa P 17f7bd3caba1497985a5d062a28ec8b9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:40:31.667394 27232 heartbeater.cc:503] Master 127.18.211.126:41149 was elected leader, sending a full tablet report...
I20260928 06:40:31.667726 27261 raft_consensus.cc:348] T 4b44d9339a83464eb45ddad7182dc9fa P 0007b185a0014c199414c8c0607806f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } } peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } }
I20260928 06:40:31.667845 27261 raft_consensus.cc:374] T 4b44d9339a83464eb45ddad7182dc9fa P 0007b185a0014c199414c8c0607806f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:31.667876 27261 raft_consensus.cc:729] T 4b44d9339a83464eb45ddad7182dc9fa P 0007b185a0014c199414c8c0607806f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0007b185a0014c199414c8c0607806f1, State: Initialized, Role: FOLLOWER
I20260928 06:40:31.668025 27261 consensus_queue.cc:260] T 4b44d9339a83464eb45ddad7182dc9fa P 0007b185a0014c199414c8c0607806f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } } peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } }
I20260928 06:40:31.667969 27262 raft_consensus.cc:348] T 4b44d9339a83464eb45ddad7182dc9fa P f034fbd6323b41a28e36c21af4b7f475 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } } peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } }
I20260928 06:40:31.668107 27262 raft_consensus.cc:374] T 4b44d9339a83464eb45ddad7182dc9fa P f034fbd6323b41a28e36c21af4b7f475 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:31.668174 27262 raft_consensus.cc:729] T 4b44d9339a83464eb45ddad7182dc9fa P f034fbd6323b41a28e36c21af4b7f475 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f034fbd6323b41a28e36c21af4b7f475, State: Initialized, Role: FOLLOWER
I20260928 06:40:31.668298 27262 consensus_queue.cc:260] T 4b44d9339a83464eb45ddad7182dc9fa P f034fbd6323b41a28e36c21af4b7f475 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } } peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } }
I20260928 06:40:31.668368 27092 heartbeater.cc:503] Master 127.18.211.126:41149 was elected leader, sending a full tablet report...
I20260928 06:40:31.668267 27261 ts_tablet_manager.cc:1428] T 4b44d9339a83464eb45ddad7182dc9fa P 0007b185a0014c199414c8c0607806f1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:40:31.668553 27262 ts_tablet_manager.cc:1428] T 4b44d9339a83464eb45ddad7182dc9fa P f034fbd6323b41a28e36c21af4b7f475: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:40:31.668599 26952 heartbeater.cc:503] Master 127.18.211.126:41149 was elected leader, sending a full tablet report...
I20260928 06:40:31.671281 27266 raft_consensus.cc:482] T 4b44d9339a83464eb45ddad7182dc9fa P 17f7bd3caba1497985a5d062a28ec8b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:40:31.671358 27266 raft_consensus.cc:504] T 4b44d9339a83464eb45ddad7182dc9fa P 17f7bd3caba1497985a5d062a28ec8b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } } peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } }
I20260928 06:40:31.671615 27266 leader_election.cc:288] T 4b44d9339a83464eb45ddad7182dc9fa P 17f7bd3caba1497985a5d062a28ec8b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f034fbd6323b41a28e36c21af4b7f475 (127.18.211.65:45105), 0007b185a0014c199414c8c0607806f1 (127.18.211.66:36675)
I20260928 06:40:31.675247 27259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:31.671807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:40403 (local address 127.18.211.65:45105)
0928 06:40:31.671842 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:31.671844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:31.671879 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:40:31.671923 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:31.671924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:31.671932 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:31.671978 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:31.672095 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.672890 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.674347 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.674877 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.674942 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:31.674950 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:31.675142 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 06:40:31.675179 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 06:40:31.675248 27125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:31.671765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:45105 (local address 127.18.211.67:40403)
0928 06:40:31.671825 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:31.671837 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:31.671868 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:31.671982 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:31.671982 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:31.671984 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:31.672040 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:31.672042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.672959 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.672960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:31.674226 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:31.674229 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.674941 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.674942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:31.675037 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:31.675041 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:40:31.675132 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 06:40:31.675177 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":59}
I20260928 06:40:31.675575 26906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b44d9339a83464eb45ddad7182dc9fa" candidate_uuid: "17f7bd3caba1497985a5d062a28ec8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f034fbd6323b41a28e36c21af4b7f475" is_pre_election: true
I20260928 06:40:31.675664 27257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:31.671848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:55279 (local address 127.18.211.66:36675)
0928 06:40:31.671882 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:31.671884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:31.671977 (+ 93us) server_negotiation.cc:407] Connection header received
0928 06:40:31.672013 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:31.672014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:31.672021 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:31.672061 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:31.672167 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.672873 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.674307 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.674871 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.674942 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:31.674950 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:31.675578 (+ 628us) server_negotiation.cc:299] Negotiation successful
0928 06:40:31.675617 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:40:31.675674 27269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:31.671766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:36675 (local address 127.18.211.67:55279)
0928 06:40:31.671919 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:31.671927 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:31.671972 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:31.672063 (+ 91us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:31.672063 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:31.672064 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:31.672114 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:31.672115 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.672959 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.672960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:31.674226 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:31.674229 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.675446 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.675447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:31.675514 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:31.675518 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:40:31.675559 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 06:40:31.675593 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":70,"thread_start_us":80,"threads_started":1}
I20260928 06:40:31.675724 26906 raft_consensus.cc:2454] T 4b44d9339a83464eb45ddad7182dc9fa P f034fbd6323b41a28e36c21af4b7f475 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17f7bd3caba1497985a5d062a28ec8b9 in term 0.
I20260928 06:40:31.675948 27118 leader_election.cc:302] T 4b44d9339a83464eb45ddad7182dc9fa P 17f7bd3caba1497985a5d062a28ec8b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17f7bd3caba1497985a5d062a28ec8b9, f034fbd6323b41a28e36c21af4b7f475; no voters:
I20260928 06:40:31.676069 27266 raft_consensus.cc:2781] T 4b44d9339a83464eb45ddad7182dc9fa P 17f7bd3caba1497985a5d062a28ec8b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:40:31.676043 27046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b44d9339a83464eb45ddad7182dc9fa" candidate_uuid: "17f7bd3caba1497985a5d062a28ec8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0007b185a0014c199414c8c0607806f1" is_pre_election: true
I20260928 06:40:31.676116 27266 raft_consensus.cc:482] T 4b44d9339a83464eb45ddad7182dc9fa P 17f7bd3caba1497985a5d062a28ec8b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:40:31.676131 27266 raft_consensus.cc:3037] T 4b44d9339a83464eb45ddad7182dc9fa P 17f7bd3caba1497985a5d062a28ec8b9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:31.676167 27046 raft_consensus.cc:2454] T 4b44d9339a83464eb45ddad7182dc9fa P 0007b185a0014c199414c8c0607806f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17f7bd3caba1497985a5d062a28ec8b9 in term 0.
I20260928 06:40:31.677007 27266 raft_consensus.cc:504] T 4b44d9339a83464eb45ddad7182dc9fa P 17f7bd3caba1497985a5d062a28ec8b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } } peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } }
I20260928 06:40:31.677126 27266 leader_election.cc:288] T 4b44d9339a83464eb45ddad7182dc9fa P 17f7bd3caba1497985a5d062a28ec8b9 [CANDIDATE]: Term 1 election: Requested vote from peers f034fbd6323b41a28e36c21af4b7f475 (127.18.211.65:45105), 0007b185a0014c199414c8c0607806f1 (127.18.211.66:36675)
I20260928 06:40:31.677241 26906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b44d9339a83464eb45ddad7182dc9fa" candidate_uuid: "17f7bd3caba1497985a5d062a28ec8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f034fbd6323b41a28e36c21af4b7f475"
I20260928 06:40:31.677290 27046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b44d9339a83464eb45ddad7182dc9fa" candidate_uuid: "17f7bd3caba1497985a5d062a28ec8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0007b185a0014c199414c8c0607806f1"
I20260928 06:40:31.677314 26906 raft_consensus.cc:3037] T 4b44d9339a83464eb45ddad7182dc9fa P f034fbd6323b41a28e36c21af4b7f475 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:31.677354 27046 raft_consensus.cc:3037] T 4b44d9339a83464eb45ddad7182dc9fa P 0007b185a0014c199414c8c0607806f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:31.678109 26906 raft_consensus.cc:2454] T 4b44d9339a83464eb45ddad7182dc9fa P f034fbd6323b41a28e36c21af4b7f475 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17f7bd3caba1497985a5d062a28ec8b9 in term 1.
I20260928 06:40:31.678109 27046 raft_consensus.cc:2454] T 4b44d9339a83464eb45ddad7182dc9fa P 0007b185a0014c199414c8c0607806f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17f7bd3caba1497985a5d062a28ec8b9 in term 1.
I20260928 06:40:31.678256 27118 leader_election.cc:302] T 4b44d9339a83464eb45ddad7182dc9fa P 17f7bd3caba1497985a5d062a28ec8b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17f7bd3caba1497985a5d062a28ec8b9, f034fbd6323b41a28e36c21af4b7f475; no voters:
I20260928 06:40:31.678339 27266 raft_consensus.cc:2781] T 4b44d9339a83464eb45ddad7182dc9fa P 17f7bd3caba1497985a5d062a28ec8b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:31.678479 27266 raft_consensus.cc:686] T 4b44d9339a83464eb45ddad7182dc9fa P 17f7bd3caba1497985a5d062a28ec8b9 [term 1 LEADER]: Becoming Leader. State: Replica: 17f7bd3caba1497985a5d062a28ec8b9, State: Running, Role: LEADER
I20260928 06:40:31.678558 27266 consensus_queue.cc:237] T 4b44d9339a83464eb45ddad7182dc9fa P 17f7bd3caba1497985a5d062a28ec8b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } } peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } }
I20260928 06:40:31.679345 26749 catalog_manager.cc:5495] T 4b44d9339a83464eb45ddad7182dc9fa P 17f7bd3caba1497985a5d062a28ec8b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17f7bd3caba1497985a5d062a28ec8b9 (127.18.211.67). New cstate: current_term: 1 leader_uuid: "17f7bd3caba1497985a5d062a28ec8b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } health_report { overall_health: UNKNOWN } } }
I20260928 06:40:31.685123 27258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:31.682943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58858 (local address 127.18.211.67:35041)
0928 06:40:31.682993 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:31.682995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:31.683004 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:40:31.683037 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:31.683038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:31.683051 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:31.683146 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:40:31.683240 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.683818 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.684571 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.684748 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.684798 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:31.684910 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:40:31.684995 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:40:31.685046 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:31.685064 (+ 18us) server_negotiation.cc:299] Negotiation successful
0928 06:40:31.685078 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 06:40:31.686633 27046 raft_consensus.cc:1261] T 4b44d9339a83464eb45ddad7182dc9fa P 0007b185a0014c199414c8c0607806f1 [term 1 FOLLOWER]: Refusing update from remote peer 17f7bd3caba1497985a5d062a28ec8b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:40:31.686640 26906 raft_consensus.cc:1261] T 4b44d9339a83464eb45ddad7182dc9fa P f034fbd6323b41a28e36c21af4b7f475 [term 1 FOLLOWER]: Refusing update from remote peer 17f7bd3caba1497985a5d062a28ec8b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:40:31.686885 27266 consensus_queue.cc:1035] T 4b44d9339a83464eb45ddad7182dc9fa P 17f7bd3caba1497985a5d062a28ec8b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:40:31.687011 27270 consensus_queue.cc:1035] T 4b44d9339a83464eb45ddad7182dc9fa P 17f7bd3caba1497985a5d062a28ec8b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:40:31.689668 27274 mvcc.cc:205] Tried to move back new op lower bound from 7334205979386327040 to 7334205979356090368. Current Snapshot: MvccSnapshot[applied={T|T < 7334205979386327040}]
I20260928 06:40:31.689675 27275 mvcc.cc:205] Tried to move back new op lower bound from 7334205979386327040 to 7334205979356090368. Current Snapshot: MvccSnapshot[applied={T|T < 7334205979386327040}]
I20260928 06:40:31.690976 26748 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:40:31.691051 26748 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:40:31.691705 26748 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29696cea0aef44818a5a20643af487e8: Sending DeleteTablet for 3 replicas of tablet 4b44d9339a83464eb45ddad7182dc9fa
I20260928 06:40:31.692005 26886 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b44d9339a83464eb45ddad7182dc9fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:31 UTC) from {username='oryx', principal='oryx/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:45992
I20260928 06:40:31.691992 27026 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b44d9339a83464eb45ddad7182dc9fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:31 UTC) from {username='oryx', principal='oryx/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:60706
I20260928 06:40:31.692021 27166 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b44d9339a83464eb45ddad7182dc9fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:31 UTC) from {username='oryx', principal='oryx/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:58852
I20260928 06:40:31.692225 27284 tablet_replica.cc:330] T 4b44d9339a83464eb45ddad7182dc9fa P 17f7bd3caba1497985a5d062a28ec8b9: stopping tablet replica
I20260928 06:40:31.692220 26748 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36286:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 06:40:31.692348 27284 raft_consensus.cc:2229] T 4b44d9339a83464eb45ddad7182dc9fa P 17f7bd3caba1497985a5d062a28ec8b9 [term 1 LEADER]: Raft consensus shutting down.
W20260928 06:40:31.692363 26748 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:40:31.692438 27282 tablet_replica.cc:330] T 4b44d9339a83464eb45ddad7182dc9fa P f034fbd6323b41a28e36c21af4b7f475: stopping tablet replica
I20260928 06:40:31.692456 27284 raft_consensus.cc:2258] T 4b44d9339a83464eb45ddad7182dc9fa P 17f7bd3caba1497985a5d062a28ec8b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:31.692516 27282 raft_consensus.cc:2229] T 4b44d9339a83464eb45ddad7182dc9fa P f034fbd6323b41a28e36c21af4b7f475 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:40:31.692610 27282 raft_consensus.cc:2258] T 4b44d9339a83464eb45ddad7182dc9fa P f034fbd6323b41a28e36c21af4b7f475 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:31.692957 27283 tablet_replica.cc:330] T 4b44d9339a83464eb45ddad7182dc9fa P 0007b185a0014c199414c8c0607806f1: stopping tablet replica
I20260928 06:40:31.693084 27283 raft_consensus.cc:2229] T 4b44d9339a83464eb45ddad7182dc9fa P 0007b185a0014c199414c8c0607806f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:40:31.693094 27282 ts_tablet_manager.cc:1905] T 4b44d9339a83464eb45ddad7182dc9fa P f034fbd6323b41a28e36c21af4b7f475: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:31.693606 27283 raft_consensus.cc:2258] T 4b44d9339a83464eb45ddad7182dc9fa P 0007b185a0014c199414c8c0607806f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:31.693771 27026 tablet_service.cc:1467] Processing CreateTablet for tablet 30e8ffe78ad44cb8a1e046011e5225f2 (DEFAULT_TABLE table=test-table [id=513d4d7a2a034fdbb54b536afad4aff8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:31.693877 27026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30e8ffe78ad44cb8a1e046011e5225f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:31.694772 27261 tablet_bootstrap.cc:492] T 30e8ffe78ad44cb8a1e046011e5225f2 P 0007b185a0014c199414c8c0607806f1: Bootstrap starting.
I20260928 06:40:31.694890 27282 ts_tablet_manager.cc:1918] T 4b44d9339a83464eb45ddad7182dc9fa P f034fbd6323b41a28e36c21af4b7f475: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:40:31.694948 27282 log.cc:1196] T 4b44d9339a83464eb45ddad7182dc9fa P f034fbd6323b41a28e36c21af4b7f475: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-0/wal/wals/4b44d9339a83464eb45ddad7182dc9fa
I20260928 06:40:31.695150 27166 tablet_service.cc:1467] Processing CreateTablet for tablet 30e8ffe78ad44cb8a1e046011e5225f2 (DEFAULT_TABLE table=test-table [id=513d4d7a2a034fdbb54b536afad4aff8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:31.695214 27261 tablet_bootstrap.cc:654] T 30e8ffe78ad44cb8a1e046011e5225f2 P 0007b185a0014c199414c8c0607806f1: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:31.695216 27282 ts_tablet_manager.cc:1939] T 4b44d9339a83464eb45ddad7182dc9fa P f034fbd6323b41a28e36c21af4b7f475: Deleting consensus metadata
I20260928 06:40:31.695243 27166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30e8ffe78ad44cb8a1e046011e5225f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:31.695693 27284 ts_tablet_manager.cc:1905] T 4b44d9339a83464eb45ddad7182dc9fa P 17f7bd3caba1497985a5d062a28ec8b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:31.695683 26886 tablet_service.cc:1467] Processing CreateTablet for tablet 30e8ffe78ad44cb8a1e046011e5225f2 (DEFAULT_TABLE table=test-table [id=513d4d7a2a034fdbb54b536afad4aff8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:31.695782 26886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30e8ffe78ad44cb8a1e046011e5225f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:31.695955 27283 ts_tablet_manager.cc:1905] T 4b44d9339a83464eb45ddad7182dc9fa P 0007b185a0014c199414c8c0607806f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:31.696092 27261 tablet_bootstrap.cc:492] T 30e8ffe78ad44cb8a1e046011e5225f2 P 0007b185a0014c199414c8c0607806f1: No bootstrap required, opened a new log
I20260928 06:40:31.696157 27261 ts_tablet_manager.cc:1397] T 30e8ffe78ad44cb8a1e046011e5225f2 P 0007b185a0014c199414c8c0607806f1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:31.696310 27261 raft_consensus.cc:348] T 30e8ffe78ad44cb8a1e046011e5225f2 P 0007b185a0014c199414c8c0607806f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } }
I20260928 06:40:31.696370 27261 raft_consensus.cc:374] T 30e8ffe78ad44cb8a1e046011e5225f2 P 0007b185a0014c199414c8c0607806f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:31.696389 27261 raft_consensus.cc:729] T 30e8ffe78ad44cb8a1e046011e5225f2 P 0007b185a0014c199414c8c0607806f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0007b185a0014c199414c8c0607806f1, State: Initialized, Role: FOLLOWER
I20260928 06:40:31.696434 27261 consensus_queue.cc:260] T 30e8ffe78ad44cb8a1e046011e5225f2 P 0007b185a0014c199414c8c0607806f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } }
I20260928 06:40:31.696524 27261 ts_tablet_manager.cc:1428] T 30e8ffe78ad44cb8a1e046011e5225f2 P 0007b185a0014c199414c8c0607806f1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:40:31.696563 26734 catalog_manager.cc:4841] TS f034fbd6323b41a28e36c21af4b7f475 (127.18.211.65:45105): tablet 4b44d9339a83464eb45ddad7182dc9fa (table test-table [id=f1db164cdcdf4a66be4e2d659e0a55c0]) successfully deleted
I20260928 06:40:31.696877 27262 tablet_bootstrap.cc:492] T 30e8ffe78ad44cb8a1e046011e5225f2 P f034fbd6323b41a28e36c21af4b7f475: Bootstrap starting.
I20260928 06:40:31.697222 27262 tablet_bootstrap.cc:654] T 30e8ffe78ad44cb8a1e046011e5225f2 P f034fbd6323b41a28e36c21af4b7f475: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:31.697302 27283 ts_tablet_manager.cc:1918] T 4b44d9339a83464eb45ddad7182dc9fa P 0007b185a0014c199414c8c0607806f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:40:31.697371 27283 log.cc:1196] T 4b44d9339a83464eb45ddad7182dc9fa P 0007b185a0014c199414c8c0607806f1: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-1/wal/wals/4b44d9339a83464eb45ddad7182dc9fa
I20260928 06:40:31.697608 27283 ts_tablet_manager.cc:1939] T 4b44d9339a83464eb45ddad7182dc9fa P 0007b185a0014c199414c8c0607806f1: Deleting consensus metadata
I20260928 06:40:31.696480 27260 tablet_bootstrap.cc:492] T 30e8ffe78ad44cb8a1e046011e5225f2 P 17f7bd3caba1497985a5d062a28ec8b9: Bootstrap starting.
I20260928 06:40:31.697744 27262 tablet_bootstrap.cc:492] T 30e8ffe78ad44cb8a1e046011e5225f2 P f034fbd6323b41a28e36c21af4b7f475: No bootstrap required, opened a new log
I20260928 06:40:31.697779 27262 ts_tablet_manager.cc:1397] T 30e8ffe78ad44cb8a1e046011e5225f2 P f034fbd6323b41a28e36c21af4b7f475: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:31.697916 27262 raft_consensus.cc:348] T 30e8ffe78ad44cb8a1e046011e5225f2 P f034fbd6323b41a28e36c21af4b7f475 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } }
I20260928 06:40:31.697983 27262 raft_consensus.cc:374] T 30e8ffe78ad44cb8a1e046011e5225f2 P f034fbd6323b41a28e36c21af4b7f475 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:31.698001 27262 raft_consensus.cc:729] T 30e8ffe78ad44cb8a1e046011e5225f2 P f034fbd6323b41a28e36c21af4b7f475 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f034fbd6323b41a28e36c21af4b7f475, State: Initialized, Role: FOLLOWER
I20260928 06:40:31.698036 27262 consensus_queue.cc:260] T 30e8ffe78ad44cb8a1e046011e5225f2 P f034fbd6323b41a28e36c21af4b7f475 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } }
I20260928 06:40:31.698103 27262 ts_tablet_manager.cc:1428] T 30e8ffe78ad44cb8a1e046011e5225f2 P f034fbd6323b41a28e36c21af4b7f475: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:31.698204 27260 tablet_bootstrap.cc:654] T 30e8ffe78ad44cb8a1e046011e5225f2 P 17f7bd3caba1497985a5d062a28ec8b9: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:31.698984 27260 tablet_bootstrap.cc:492] T 30e8ffe78ad44cb8a1e046011e5225f2 P 17f7bd3caba1497985a5d062a28ec8b9: No bootstrap required, opened a new log
I20260928 06:40:31.699023 27260 ts_tablet_manager.cc:1397] T 30e8ffe78ad44cb8a1e046011e5225f2 P 17f7bd3caba1497985a5d062a28ec8b9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 06:40:31.699051 27284 ts_tablet_manager.cc:1918] T 4b44d9339a83464eb45ddad7182dc9fa P 17f7bd3caba1497985a5d062a28ec8b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:40:31.699196 27284 log.cc:1196] T 4b44d9339a83464eb45ddad7182dc9fa P 17f7bd3caba1497985a5d062a28ec8b9: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-2/wal/wals/4b44d9339a83464eb45ddad7182dc9fa
I20260928 06:40:31.699173 27260 raft_consensus.cc:348] T 30e8ffe78ad44cb8a1e046011e5225f2 P 17f7bd3caba1497985a5d062a28ec8b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } }
I20260928 06:40:31.699294 27260 raft_consensus.cc:374] T 30e8ffe78ad44cb8a1e046011e5225f2 P 17f7bd3caba1497985a5d062a28ec8b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:31.699318 27260 raft_consensus.cc:729] T 30e8ffe78ad44cb8a1e046011e5225f2 P 17f7bd3caba1497985a5d062a28ec8b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17f7bd3caba1497985a5d062a28ec8b9, State: Initialized, Role: FOLLOWER
I20260928 06:40:31.699362 27260 consensus_queue.cc:260] T 30e8ffe78ad44cb8a1e046011e5225f2 P 17f7bd3caba1497985a5d062a28ec8b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } }
I20260928 06:40:31.699385 26733 catalog_manager.cc:4841] TS 0007b185a0014c199414c8c0607806f1 (127.18.211.66:36675): tablet 4b44d9339a83464eb45ddad7182dc9fa (table test-table [id=f1db164cdcdf4a66be4e2d659e0a55c0]) successfully deleted
I20260928 06:40:31.699445 27260 ts_tablet_manager.cc:1428] T 30e8ffe78ad44cb8a1e046011e5225f2 P 17f7bd3caba1497985a5d062a28ec8b9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:31.699488 27284 ts_tablet_manager.cc:1939] T 4b44d9339a83464eb45ddad7182dc9fa P 17f7bd3caba1497985a5d062a28ec8b9: Deleting consensus metadata
I20260928 06:40:31.699774 26735 catalog_manager.cc:4841] TS 17f7bd3caba1497985a5d062a28ec8b9 (127.18.211.67:35041): tablet 4b44d9339a83464eb45ddad7182dc9fa (table test-table [id=f1db164cdcdf4a66be4e2d659e0a55c0]) successfully deleted
W20260928 06:40:31.717605 27093 tablet.cc:2369] T 30e8ffe78ad44cb8a1e046011e5225f2 P 0007b185a0014c199414c8c0607806f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:40:31.764058 27266 raft_consensus.cc:482] T 30e8ffe78ad44cb8a1e046011e5225f2 P 17f7bd3caba1497985a5d062a28ec8b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:40:31.764161 27266 raft_consensus.cc:504] T 30e8ffe78ad44cb8a1e046011e5225f2 P 17f7bd3caba1497985a5d062a28ec8b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } }
I20260928 06:40:31.764353 27266 leader_election.cc:288] T 30e8ffe78ad44cb8a1e046011e5225f2 P 17f7bd3caba1497985a5d062a28ec8b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f034fbd6323b41a28e36c21af4b7f475 (127.18.211.65:45105), 0007b185a0014c199414c8c0607806f1 (127.18.211.66:36675)
I20260928 06:40:31.764596 26906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e8ffe78ad44cb8a1e046011e5225f2" candidate_uuid: "17f7bd3caba1497985a5d062a28ec8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f034fbd6323b41a28e36c21af4b7f475" is_pre_election: true
I20260928 06:40:31.764591 27046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e8ffe78ad44cb8a1e046011e5225f2" candidate_uuid: "17f7bd3caba1497985a5d062a28ec8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0007b185a0014c199414c8c0607806f1" is_pre_election: true
I20260928 06:40:31.764698 27046 raft_consensus.cc:2454] T 30e8ffe78ad44cb8a1e046011e5225f2 P 0007b185a0014c199414c8c0607806f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17f7bd3caba1497985a5d062a28ec8b9 in term 0.
I20260928 06:40:31.764715 26906 raft_consensus.cc:2454] T 30e8ffe78ad44cb8a1e046011e5225f2 P f034fbd6323b41a28e36c21af4b7f475 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17f7bd3caba1497985a5d062a28ec8b9 in term 0.
I20260928 06:40:31.764864 27117 leader_election.cc:302] T 30e8ffe78ad44cb8a1e046011e5225f2 P 17f7bd3caba1497985a5d062a28ec8b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0007b185a0014c199414c8c0607806f1, 17f7bd3caba1497985a5d062a28ec8b9; no voters:
I20260928 06:40:31.764957 27266 raft_consensus.cc:2781] T 30e8ffe78ad44cb8a1e046011e5225f2 P 17f7bd3caba1497985a5d062a28ec8b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:40:31.764997 27266 raft_consensus.cc:482] T 30e8ffe78ad44cb8a1e046011e5225f2 P 17f7bd3caba1497985a5d062a28ec8b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:40:31.765009 27266 raft_consensus.cc:3037] T 30e8ffe78ad44cb8a1e046011e5225f2 P 17f7bd3caba1497985a5d062a28ec8b9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:31.765679 27266 raft_consensus.cc:504] T 30e8ffe78ad44cb8a1e046011e5225f2 P 17f7bd3caba1497985a5d062a28ec8b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } }
I20260928 06:40:31.765794 27266 leader_election.cc:288] T 30e8ffe78ad44cb8a1e046011e5225f2 P 17f7bd3caba1497985a5d062a28ec8b9 [CANDIDATE]: Term 1 election: Requested vote from peers f034fbd6323b41a28e36c21af4b7f475 (127.18.211.65:45105), 0007b185a0014c199414c8c0607806f1 (127.18.211.66:36675)
I20260928 06:40:31.765966 27046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e8ffe78ad44cb8a1e046011e5225f2" candidate_uuid: "17f7bd3caba1497985a5d062a28ec8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0007b185a0014c199414c8c0607806f1"
I20260928 06:40:31.766036 27046 raft_consensus.cc:3037] T 30e8ffe78ad44cb8a1e046011e5225f2 P 0007b185a0014c199414c8c0607806f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:31.766009 26906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e8ffe78ad44cb8a1e046011e5225f2" candidate_uuid: "17f7bd3caba1497985a5d062a28ec8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f034fbd6323b41a28e36c21af4b7f475"
I20260928 06:40:31.766113 26906 raft_consensus.cc:3037] T 30e8ffe78ad44cb8a1e046011e5225f2 P f034fbd6323b41a28e36c21af4b7f475 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:31.766618 27046 raft_consensus.cc:2454] T 30e8ffe78ad44cb8a1e046011e5225f2 P 0007b185a0014c199414c8c0607806f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17f7bd3caba1497985a5d062a28ec8b9 in term 1.
I20260928 06:40:31.766770 27117 leader_election.cc:302] T 30e8ffe78ad44cb8a1e046011e5225f2 P 17f7bd3caba1497985a5d062a28ec8b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0007b185a0014c199414c8c0607806f1, 17f7bd3caba1497985a5d062a28ec8b9; no voters:
I20260928 06:40:31.766851 27266 raft_consensus.cc:2781] T 30e8ffe78ad44cb8a1e046011e5225f2 P 17f7bd3caba1497985a5d062a28ec8b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:31.766892 27266 raft_consensus.cc:686] T 30e8ffe78ad44cb8a1e046011e5225f2 P 17f7bd3caba1497985a5d062a28ec8b9 [term 1 LEADER]: Becoming Leader. State: Replica: 17f7bd3caba1497985a5d062a28ec8b9, State: Running, Role: LEADER
I20260928 06:40:31.766912 26906 raft_consensus.cc:2454] T 30e8ffe78ad44cb8a1e046011e5225f2 P f034fbd6323b41a28e36c21af4b7f475 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17f7bd3caba1497985a5d062a28ec8b9 in term 1.
I20260928 06:40:31.766944 27266 consensus_queue.cc:237] T 30e8ffe78ad44cb8a1e046011e5225f2 P 17f7bd3caba1497985a5d062a28ec8b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } }
I20260928 06:40:31.767551 26748 catalog_manager.cc:5495] T 30e8ffe78ad44cb8a1e046011e5225f2 P 17f7bd3caba1497985a5d062a28ec8b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17f7bd3caba1497985a5d062a28ec8b9 (127.18.211.67). New cstate: current_term: 1 leader_uuid: "17f7bd3caba1497985a5d062a28ec8b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } health_report { overall_health: HEALTHY } } }
W20260928 06:40:31.821296 26953 tablet.cc:2369] T 30e8ffe78ad44cb8a1e046011e5225f2 P f034fbd6323b41a28e36c21af4b7f475: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 06:40:31 dist-test-slave-b28r krb5kdc[26704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577631, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.126@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM
I20260928 06:40:31.834376 27294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:31.827831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:41149 (local address 127.0.0.1:36290)
0928 06:40:31.828018 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:31.828029 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:31.828102 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:31.828229 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:31.828230 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:31.828540 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:31.828588 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:31.828590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.829263 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.829264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:31.829791 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:31.829792 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.830159 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.830160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:31.830257 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:31.830813 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:31.830817 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:31.832242 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:31.833545 (+ 1303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:31.833552 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:31.833554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:31.833739 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:31.833900 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:31.833900 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:31.833900 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:31.833953 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:31.834123 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:31.834128 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:31.834165 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 06:40:31.834209 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 06:40:31.834303 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":104,"threads_started":1}
I20260928 06:40:31.834450 26798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:31.827918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36290 (local address 127.18.211.126:41149)
0928 06:40:31.827978 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:31.827981 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:31.828084 (+ 103us) server_negotiation.cc:407] Connection header received
0928 06:40:31.828165 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:31.828166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:31.828176 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:31.828214 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:31.828660 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.829194 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.829873 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:31.830056 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:31.830119 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:31.832356 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:31.832358 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:31.832358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:31.832365 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:31.833477 (+ 1112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:31.833808 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:31.833808 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:31.833809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:31.833855 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:31.833994 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:31.833995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:31.833995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:31.834083 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:31.834126 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:31.834230 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 06:40:31.834354 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":33}
I20260928 06:40:31.836201 26748 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:36290:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 06:40:31.836388 26748 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:40:31.837826 26886 tablet_service.cc:1467] Processing CreateTablet for tablet 52d03c57d2cb4997b0196888e8c9e47d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aff403a80d55415587ede112f4735b6e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:40:31.837854 27166 tablet_service.cc:1467] Processing CreateTablet for tablet 52d03c57d2cb4997b0196888e8c9e47d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aff403a80d55415587ede112f4735b6e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:40:31.837988 26886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52d03c57d2cb4997b0196888e8c9e47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:31.837988 27166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52d03c57d2cb4997b0196888e8c9e47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:31.839493 27262 tablet_bootstrap.cc:492] T 52d03c57d2cb4997b0196888e8c9e47d P f034fbd6323b41a28e36c21af4b7f475: Bootstrap starting.
I20260928 06:40:31.839493 27260 tablet_bootstrap.cc:492] T 52d03c57d2cb4997b0196888e8c9e47d P 17f7bd3caba1497985a5d062a28ec8b9: Bootstrap starting.
I20260928 06:40:31.839556 27026 tablet_service.cc:1467] Processing CreateTablet for tablet 52d03c57d2cb4997b0196888e8c9e47d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=aff403a80d55415587ede112f4735b6e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 06:40:31.839704 27026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52d03c57d2cb4997b0196888e8c9e47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:31.839883 27262 tablet_bootstrap.cc:654] T 52d03c57d2cb4997b0196888e8c9e47d P f034fbd6323b41a28e36c21af4b7f475: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:31.839883 27260 tablet_bootstrap.cc:654] T 52d03c57d2cb4997b0196888e8c9e47d P 17f7bd3caba1497985a5d062a28ec8b9: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:31.840939 27261 tablet_bootstrap.cc:492] T 52d03c57d2cb4997b0196888e8c9e47d P 0007b185a0014c199414c8c0607806f1: Bootstrap starting.
I20260928 06:40:31.841171 27260 tablet_bootstrap.cc:492] T 52d03c57d2cb4997b0196888e8c9e47d P 17f7bd3caba1497985a5d062a28ec8b9: No bootstrap required, opened a new log
I20260928 06:40:31.841217 27260 ts_tablet_manager.cc:1397] T 52d03c57d2cb4997b0196888e8c9e47d P 17f7bd3caba1497985a5d062a28ec8b9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 06:40:31.841336 27261 tablet_bootstrap.cc:654] T 52d03c57d2cb4997b0196888e8c9e47d P 0007b185a0014c199414c8c0607806f1: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:31.841360 27260 raft_consensus.cc:348] T 52d03c57d2cb4997b0196888e8c9e47d P 17f7bd3caba1497985a5d062a28ec8b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } }
I20260928 06:40:31.841413 27260 raft_consensus.cc:374] T 52d03c57d2cb4997b0196888e8c9e47d P 17f7bd3caba1497985a5d062a28ec8b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:31.841425 27260 raft_consensus.cc:729] T 52d03c57d2cb4997b0196888e8c9e47d P 17f7bd3caba1497985a5d062a28ec8b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17f7bd3caba1497985a5d062a28ec8b9, State: Initialized, Role: FOLLOWER
I20260928 06:40:31.841457 27260 consensus_queue.cc:260] T 52d03c57d2cb4997b0196888e8c9e47d P 17f7bd3caba1497985a5d062a28ec8b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } }
I20260928 06:40:31.841530 27260 ts_tablet_manager.cc:1428] T 52d03c57d2cb4997b0196888e8c9e47d P 17f7bd3caba1497985a5d062a28ec8b9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:31.841568 27266 tablet_replica.cc:439] T 52d03c57d2cb4997b0196888e8c9e47d P 17f7bd3caba1497985a5d062a28ec8b9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } } }
I20260928 06:40:31.841635 27266 tablet_replica.cc:442] T 52d03c57d2cb4997b0196888e8c9e47d P 17f7bd3caba1497985a5d062a28ec8b9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:40:31.841900 27262 tablet_bootstrap.cc:492] T 52d03c57d2cb4997b0196888e8c9e47d P f034fbd6323b41a28e36c21af4b7f475: No bootstrap required, opened a new log
I20260928 06:40:31.841951 27262 ts_tablet_manager.cc:1397] T 52d03c57d2cb4997b0196888e8c9e47d P f034fbd6323b41a28e36c21af4b7f475: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:40:31.842077 27261 tablet_bootstrap.cc:492] T 52d03c57d2cb4997b0196888e8c9e47d P 0007b185a0014c199414c8c0607806f1: No bootstrap required, opened a new log
I20260928 06:40:31.842088 27262 raft_consensus.cc:348] T 52d03c57d2cb4997b0196888e8c9e47d P f034fbd6323b41a28e36c21af4b7f475 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } }
I20260928 06:40:31.842128 27261 ts_tablet_manager.cc:1397] T 52d03c57d2cb4997b0196888e8c9e47d P 0007b185a0014c199414c8c0607806f1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:31.842161 27262 raft_consensus.cc:374] T 52d03c57d2cb4997b0196888e8c9e47d P f034fbd6323b41a28e36c21af4b7f475 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:31.842188 27262 raft_consensus.cc:729] T 52d03c57d2cb4997b0196888e8c9e47d P f034fbd6323b41a28e36c21af4b7f475 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f034fbd6323b41a28e36c21af4b7f475, State: Initialized, Role: FOLLOWER
I20260928 06:40:31.842276 27261 raft_consensus.cc:348] T 52d03c57d2cb4997b0196888e8c9e47d P 0007b185a0014c199414c8c0607806f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } }
I20260928 06:40:31.842305 27262 consensus_queue.cc:260] T 52d03c57d2cb4997b0196888e8c9e47d P f034fbd6323b41a28e36c21af4b7f475 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } }
I20260928 06:40:31.842358 27261 raft_consensus.cc:374] T 52d03c57d2cb4997b0196888e8c9e47d P 0007b185a0014c199414c8c0607806f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:31.842384 27261 raft_consensus.cc:729] T 52d03c57d2cb4997b0196888e8c9e47d P 0007b185a0014c199414c8c0607806f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0007b185a0014c199414c8c0607806f1, State: Initialized, Role: FOLLOWER
I20260928 06:40:31.842413 27262 ts_tablet_manager.cc:1428] T 52d03c57d2cb4997b0196888e8c9e47d P f034fbd6323b41a28e36c21af4b7f475: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260928 06:40:31.842422 27261 consensus_queue.cc:260] T 52d03c57d2cb4997b0196888e8c9e47d P 0007b185a0014c199414c8c0607806f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } }
I20260928 06:40:31.842509 27261 ts_tablet_manager.cc:1428] T 52d03c57d2cb4997b0196888e8c9e47d P 0007b185a0014c199414c8c0607806f1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:31.842530 27268 tablet_replica.cc:439] T 52d03c57d2cb4997b0196888e8c9e47d P f034fbd6323b41a28e36c21af4b7f475: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } } }
I20260928 06:40:31.842617 27268 tablet_replica.cc:442] T 52d03c57d2cb4997b0196888e8c9e47d P f034fbd6323b41a28e36c21af4b7f475: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:40:31.842814 27267 tablet_replica.cc:439] T 52d03c57d2cb4997b0196888e8c9e47d P 0007b185a0014c199414c8c0607806f1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } } }
I20260928 06:40:31.842890 27267 tablet_replica.cc:442] T 52d03c57d2cb4997b0196888e8c9e47d P 0007b185a0014c199414c8c0607806f1: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 06:40:31.863808 27233 tablet.cc:2369] T 30e8ffe78ad44cb8a1e046011e5225f2 P 17f7bd3caba1497985a5d062a28ec8b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:40:32.138067 27268 raft_consensus.cc:482] T 52d03c57d2cb4997b0196888e8c9e47d P f034fbd6323b41a28e36c21af4b7f475 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:40:32.138195 27268 raft_consensus.cc:504] T 52d03c57d2cb4997b0196888e8c9e47d P f034fbd6323b41a28e36c21af4b7f475 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } }
I20260928 06:40:32.138622 27268 leader_election.cc:288] T 52d03c57d2cb4997b0196888e8c9e47d P f034fbd6323b41a28e36c21af4b7f475 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0007b185a0014c199414c8c0607806f1 (127.18.211.66:36675), 17f7bd3caba1497985a5d062a28ec8b9 (127.18.211.67:35041)
I20260928 06:40:32.138823 27266 raft_consensus.cc:482] T 52d03c57d2cb4997b0196888e8c9e47d P 17f7bd3caba1497985a5d062a28ec8b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:40:32.138890 27266 raft_consensus.cc:504] T 52d03c57d2cb4997b0196888e8c9e47d P 17f7bd3caba1497985a5d062a28ec8b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } }
I20260928 06:40:32.139016 27266 leader_election.cc:288] T 52d03c57d2cb4997b0196888e8c9e47d P 17f7bd3caba1497985a5d062a28ec8b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f034fbd6323b41a28e36c21af4b7f475 (127.18.211.65:45105), 0007b185a0014c199414c8c0607806f1 (127.18.211.66:36675)
I20260928 06:40:32.139223 26906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52d03c57d2cb4997b0196888e8c9e47d" candidate_uuid: "17f7bd3caba1497985a5d062a28ec8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f034fbd6323b41a28e36c21af4b7f475" is_pre_election: true
I20260928 06:40:32.139338 26906 raft_consensus.cc:2454] T 52d03c57d2cb4997b0196888e8c9e47d P f034fbd6323b41a28e36c21af4b7f475 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17f7bd3caba1497985a5d062a28ec8b9 in term 0.
I20260928 06:40:32.139389 27046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52d03c57d2cb4997b0196888e8c9e47d" candidate_uuid: "17f7bd3caba1497985a5d062a28ec8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0007b185a0014c199414c8c0607806f1" is_pre_election: true
I20260928 06:40:32.139483 27118 leader_election.cc:302] T 52d03c57d2cb4997b0196888e8c9e47d P 17f7bd3caba1497985a5d062a28ec8b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17f7bd3caba1497985a5d062a28ec8b9, f034fbd6323b41a28e36c21af4b7f475; no voters:
I20260928 06:40:32.139499 27046 raft_consensus.cc:2454] T 52d03c57d2cb4997b0196888e8c9e47d P 0007b185a0014c199414c8c0607806f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17f7bd3caba1497985a5d062a28ec8b9 in term 0.
I20260928 06:40:32.139570 27266 raft_consensus.cc:2781] T 52d03c57d2cb4997b0196888e8c9e47d P 17f7bd3caba1497985a5d062a28ec8b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:40:32.139605 27266 raft_consensus.cc:482] T 52d03c57d2cb4997b0196888e8c9e47d P 17f7bd3caba1497985a5d062a28ec8b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:40:32.139623 27266 raft_consensus.cc:3037] T 52d03c57d2cb4997b0196888e8c9e47d P 17f7bd3caba1497985a5d062a28ec8b9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:32.140478 27266 raft_consensus.cc:504] T 52d03c57d2cb4997b0196888e8c9e47d P 17f7bd3caba1497985a5d062a28ec8b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } }
I20260928 06:40:32.140599 27266 leader_election.cc:288] T 52d03c57d2cb4997b0196888e8c9e47d P 17f7bd3caba1497985a5d062a28ec8b9 [CANDIDATE]: Term 1 election: Requested vote from peers f034fbd6323b41a28e36c21af4b7f475 (127.18.211.65:45105), 0007b185a0014c199414c8c0607806f1 (127.18.211.66:36675)
I20260928 06:40:32.140807 26906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52d03c57d2cb4997b0196888e8c9e47d" candidate_uuid: "17f7bd3caba1497985a5d062a28ec8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f034fbd6323b41a28e36c21af4b7f475"
I20260928 06:40:32.140868 26906 raft_consensus.cc:3037] T 52d03c57d2cb4997b0196888e8c9e47d P f034fbd6323b41a28e36c21af4b7f475 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:32.141599 26906 raft_consensus.cc:2454] T 52d03c57d2cb4997b0196888e8c9e47d P f034fbd6323b41a28e36c21af4b7f475 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17f7bd3caba1497985a5d062a28ec8b9 in term 1.
I20260928 06:40:32.141767 27118 leader_election.cc:302] T 52d03c57d2cb4997b0196888e8c9e47d P 17f7bd3caba1497985a5d062a28ec8b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17f7bd3caba1497985a5d062a28ec8b9, f034fbd6323b41a28e36c21af4b7f475; no voters:
I20260928 06:40:32.141929 27046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52d03c57d2cb4997b0196888e8c9e47d" candidate_uuid: "17f7bd3caba1497985a5d062a28ec8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0007b185a0014c199414c8c0607806f1"
I20260928 06:40:32.141999 27046 raft_consensus.cc:3037] T 52d03c57d2cb4997b0196888e8c9e47d P 0007b185a0014c199414c8c0607806f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:32.142704 27257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:32.139130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:60853 (local address 127.18.211.66:36675)
0928 06:40:32.139225 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:32.139227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:32.139233 (+ 6us) server_negotiation.cc:407] Connection header received
0928 06:40:32.139255 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:32.139256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:32.139264 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:32.139323 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:32.139572 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:32.140228 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:32.141750 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:32.142304 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:32.142344 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:32.142353 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:32.142599 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 06:40:32.142638 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 06:40:32.142730 27046 raft_consensus.cc:2454] T 52d03c57d2cb4997b0196888e8c9e47d P 0007b185a0014c199414c8c0607806f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17f7bd3caba1497985a5d062a28ec8b9 in term 1.
I20260928 06:40:32.142891 27299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:32.138807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:36675 (local address 127.18.211.65:60853)
0928 06:40:32.138926 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:32.138939 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:32.138999 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:32.139329 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:32.139330 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:32.139332 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:32.139413 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:32.139416 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:32.140330 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:32.140331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:32.141659 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:32.141662 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:32.142421 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:32.142421 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:32.142525 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:32.142530 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:40:32.142770 (+ 240us) client_negotiation.cc:240] Negotiation successful
0928 06:40:32.142813 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":67,"threads_started":1}
I20260928 06:40:32.143064 27258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:32.138987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:52513 (local address 127.18.211.67:35041)
0928 06:40:32.139036 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:32.139038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:32.139197 (+ 159us) server_negotiation.cc:407] Connection header received
0928 06:40:32.139223 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:32.139224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:32.139236 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:32.139290 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:32.139505 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:32.140839 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:32.142218 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:32.142709 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:32.142749 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:32.142756 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:32.142968 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 06:40:32.143009 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:40:32.143275 27298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:32.138810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:35041 (local address 127.18.211.65:52513)
0928 06:40:32.139066 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:32.139076 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:32.139153 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:32.139360 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:32.139361 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:32.139363 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:32.139429 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:32.139432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:32.141003 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:32.141003 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:32.142126 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:32.142128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:32.142820 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:32.142820 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:32.142888 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:32.142891 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:40:32.143158 (+ 267us) client_negotiation.cc:240] Negotiation successful
0928 06:40:32.143202 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":69,"threads_started":1}
I20260928 06:40:32.143381 27046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52d03c57d2cb4997b0196888e8c9e47d" candidate_uuid: "f034fbd6323b41a28e36c21af4b7f475" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0007b185a0014c199414c8c0607806f1" is_pre_election: true
I20260928 06:40:32.143476 27046 raft_consensus.cc:2379] T 52d03c57d2cb4997b0196888e8c9e47d P 0007b185a0014c199414c8c0607806f1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f034fbd6323b41a28e36c21af4b7f475 in current term 1: Already voted for candidate 17f7bd3caba1497985a5d062a28ec8b9 in this term.
I20260928 06:40:32.143508 27186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52d03c57d2cb4997b0196888e8c9e47d" candidate_uuid: "f034fbd6323b41a28e36c21af4b7f475" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17f7bd3caba1497985a5d062a28ec8b9" is_pre_election: true
I20260928 06:40:32.143615 27266 raft_consensus.cc:2781] T 52d03c57d2cb4997b0196888e8c9e47d P 17f7bd3caba1497985a5d062a28ec8b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:32.143675 27266 raft_consensus.cc:686] T 52d03c57d2cb4997b0196888e8c9e47d P 17f7bd3caba1497985a5d062a28ec8b9 [term 1 LEADER]: Becoming Leader. State: Replica: 17f7bd3caba1497985a5d062a28ec8b9, State: Running, Role: LEADER
I20260928 06:40:32.143718 27266 consensus_queue.cc:237] T 52d03c57d2cb4997b0196888e8c9e47d P 17f7bd3caba1497985a5d062a28ec8b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } }
I20260928 06:40:32.144042 27270 tablet_replica.cc:439] T 52d03c57d2cb4997b0196888e8c9e47d P 17f7bd3caba1497985a5d062a28ec8b9: TxnStatusTablet state changed. Reason: New leader 17f7bd3caba1497985a5d062a28ec8b9. Latest consensus state: current_term: 1 leader_uuid: "17f7bd3caba1497985a5d062a28ec8b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } } }
I20260928 06:40:32.144116 27270 tablet_replica.cc:442] T 52d03c57d2cb4997b0196888e8c9e47d P 17f7bd3caba1497985a5d062a28ec8b9: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:40:32.144233 27301 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 06:40:32.144376 27301 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 06:40:32.144352 26839 leader_election.cc:302] T 52d03c57d2cb4997b0196888e8c9e47d P f034fbd6323b41a28e36c21af4b7f475 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f034fbd6323b41a28e36c21af4b7f475; no voters: 0007b185a0014c199414c8c0607806f1, 17f7bd3caba1497985a5d062a28ec8b9
I20260928 06:40:32.144515 27268 raft_consensus.cc:2726] T 52d03c57d2cb4997b0196888e8c9e47d P f034fbd6323b41a28e36c21af4b7f475 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 06:40:32.144516 26749 catalog_manager.cc:5495] T 52d03c57d2cb4997b0196888e8c9e47d P 17f7bd3caba1497985a5d062a28ec8b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17f7bd3caba1497985a5d062a28ec8b9 (127.18.211.67). New cstate: current_term: 1 leader_uuid: "17f7bd3caba1497985a5d062a28ec8b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } health_report { overall_health: HEALTHY } } }
I20260928 06:40:32.194372 27258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:32.191842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58872 (local address 127.18.211.67:35041)
0928 06:40:32.191882 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:32.191884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:32.191934 (+ 50us) server_negotiation.cc:407] Connection header received
0928 06:40:32.192020 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:32.192020 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:32.192029 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:32.192075 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:40:32.192200 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:32.192651 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:32.193714 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:32.193911 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:32.193959 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:32.194157 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:40:32.194187 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:40:32.194226 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:32.194292 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 06:40:32.194313 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:40:32.194422 27294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:32.191804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:35041 (local address 127.0.0.1:58872)
0928 06:40:32.191892 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:32.191903 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:32.191955 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:32.192093 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:32.192094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:32.192095 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:40:32.192148 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:32.192150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:32.192718 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:32.192719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:32.193645 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:32.193649 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:32.193978 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:32.193979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:32.194047 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:32.194066 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:40:32.194239 (+ 173us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:40:32.194246 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 06:40:32.194289 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 06:40:32.194339 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 06:40:32.195422 27046 raft_consensus.cc:1261] T 52d03c57d2cb4997b0196888e8c9e47d P 0007b185a0014c199414c8c0607806f1 [term 1 FOLLOWER]: Refusing update from remote peer 17f7bd3caba1497985a5d062a28ec8b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:40:32.195426 26906 raft_consensus.cc:1261] T 52d03c57d2cb4997b0196888e8c9e47d P f034fbd6323b41a28e36c21af4b7f475 [term 1 FOLLOWER]: Refusing update from remote peer 17f7bd3caba1497985a5d062a28ec8b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:40:32.195618 27266 consensus_queue.cc:1035] T 52d03c57d2cb4997b0196888e8c9e47d P 17f7bd3caba1497985a5d062a28ec8b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:40:32.195871 27266 consensus_queue.cc:1035] T 52d03c57d2cb4997b0196888e8c9e47d P 17f7bd3caba1497985a5d062a28ec8b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:40:32.196529 27267 tablet_replica.cc:439] T 52d03c57d2cb4997b0196888e8c9e47d P 0007b185a0014c199414c8c0607806f1: TxnStatusTablet state changed. Reason: New leader 17f7bd3caba1497985a5d062a28ec8b9. Latest consensus state: current_term: 1 leader_uuid: "17f7bd3caba1497985a5d062a28ec8b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } } }
I20260928 06:40:32.196547 27268 tablet_replica.cc:439] T 52d03c57d2cb4997b0196888e8c9e47d P f034fbd6323b41a28e36c21af4b7f475: TxnStatusTablet state changed. Reason: New leader 17f7bd3caba1497985a5d062a28ec8b9. Latest consensus state: current_term: 1 leader_uuid: "17f7bd3caba1497985a5d062a28ec8b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } } }
I20260928 06:40:32.196626 27267 tablet_replica.cc:442] T 52d03c57d2cb4997b0196888e8c9e47d P 0007b185a0014c199414c8c0607806f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:40:32.196626 27268 tablet_replica.cc:442] T 52d03c57d2cb4997b0196888e8c9e47d P f034fbd6323b41a28e36c21af4b7f475: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:40:32.197172 27270 tablet_replica.cc:439] T 52d03c57d2cb4997b0196888e8c9e47d P 17f7bd3caba1497985a5d062a28ec8b9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "17f7bd3caba1497985a5d062a28ec8b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } } }
I20260928 06:40:32.197252 27270 tablet_replica.cc:442] T 52d03c57d2cb4997b0196888e8c9e47d P 17f7bd3caba1497985a5d062a28ec8b9: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:40:32.197507 27267 tablet_replica.cc:439] T 52d03c57d2cb4997b0196888e8c9e47d P 0007b185a0014c199414c8c0607806f1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "17f7bd3caba1497985a5d062a28ec8b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } } }
I20260928 06:40:32.197575 27267 tablet_replica.cc:442] T 52d03c57d2cb4997b0196888e8c9e47d P 0007b185a0014c199414c8c0607806f1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:40:32.197587 27270 tablet_replica.cc:439] T 52d03c57d2cb4997b0196888e8c9e47d P 17f7bd3caba1497985a5d062a28ec8b9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "17f7bd3caba1497985a5d062a28ec8b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } } }
I20260928 06:40:32.197659 27270 tablet_replica.cc:442] T 52d03c57d2cb4997b0196888e8c9e47d P 17f7bd3caba1497985a5d062a28ec8b9: This TxnStatusTablet replica's current role is: LEADER
I20260928 06:40:32.198213 27268 tablet_replica.cc:439] T 52d03c57d2cb4997b0196888e8c9e47d P f034fbd6323b41a28e36c21af4b7f475: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "17f7bd3caba1497985a5d062a28ec8b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } } }
I20260928 06:40:32.198297 27268 tablet_replica.cc:442] T 52d03c57d2cb4997b0196888e8c9e47d P f034fbd6323b41a28e36c21af4b7f475: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 06:40:32.203346 27314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:32.201190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:35041 (local address 127.18.211.67:45677)
0928 06:40:32.201299 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:32.201309 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:32.201363 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:32.201464 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:32.201465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:32.201466 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 06:40:32.201521 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:32.201522 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:32.202076 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:32.202077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:32.202636 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:32.202637 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:32.202949 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:32.202950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:32.203008 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:32.203022 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 06:40:32.203180 (+ 158us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 06:40:32.203182 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:40:32.203228 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 06:40:32.203275 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":37,"threads_started":1}
I20260928 06:40:32.203346 27258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:32.201250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:45677 (local address 127.18.211.67:35041)
0928 06:40:32.201291 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:32.201293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:32.201350 (+ 57us) server_negotiation.cc:407] Connection header received
0928 06:40:32.201403 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:32.201404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:32.201413 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:32.201461 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:40:32.201573 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:32.202016 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:32.202703 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:32.202895 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:32.202946 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:32.203103 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:40:32.203133 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:40:32.203175 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:32.203233 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 06:40:32.203258 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:40:32.205299 27046 raft_consensus.cc:1261] T 30e8ffe78ad44cb8a1e046011e5225f2 P 0007b185a0014c199414c8c0607806f1 [term 1 FOLLOWER]: Refusing update from remote peer 17f7bd3caba1497985a5d062a28ec8b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:40:32.205502 26906 raft_consensus.cc:1261] T 30e8ffe78ad44cb8a1e046011e5225f2 P f034fbd6323b41a28e36c21af4b7f475 [term 1 FOLLOWER]: Refusing update from remote peer 17f7bd3caba1497985a5d062a28ec8b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:40:32.205487 27315 consensus_queue.cc:1035] T 30e8ffe78ad44cb8a1e046011e5225f2 P 17f7bd3caba1497985a5d062a28ec8b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:40:32.205693 27315 consensus_queue.cc:1035] T 30e8ffe78ad44cb8a1e046011e5225f2 P 17f7bd3caba1497985a5d062a28ec8b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:40:32.206693 27304 mvcc.cc:205] Tried to move back new op lower bound from 7334205981511737344 to 7334205979717779456. Current Snapshot: MvccSnapshot[applied={T|T < 7334205981511737344}]
I20260928 06:40:32.216259 26749 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:40:32.216367 26749 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36286:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:40:32.216970 26749 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29696cea0aef44818a5a20643af487e8: Sending DeleteTablet for 3 replicas of tablet 30e8ffe78ad44cb8a1e046011e5225f2
I20260928 06:40:32.217195 27026 tablet_service.cc:1514] Processing DeleteTablet for tablet 30e8ffe78ad44cb8a1e046011e5225f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:32 UTC) from {username='oryx', principal='oryx/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:60706
I20260928 06:40:32.217375 26886 tablet_service.cc:1514] Processing DeleteTablet for tablet 30e8ffe78ad44cb8a1e046011e5225f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:32 UTC) from {username='oryx', principal='oryx/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:45992
I20260928 06:40:32.217568 27166 tablet_service.cc:1514] Processing DeleteTablet for tablet 30e8ffe78ad44cb8a1e046011e5225f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:32 UTC) from {username='oryx', principal='oryx/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:58852
I20260928 06:40:32.217757 27321 tablet_replica.cc:330] T 30e8ffe78ad44cb8a1e046011e5225f2 P 17f7bd3caba1497985a5d062a28ec8b9: stopping tablet replica
I20260928 06:40:32.217761 27320 tablet_replica.cc:330] T 30e8ffe78ad44cb8a1e046011e5225f2 P f034fbd6323b41a28e36c21af4b7f475: stopping tablet replica
I20260928 06:40:32.217815 27320 raft_consensus.cc:2229] T 30e8ffe78ad44cb8a1e046011e5225f2 P f034fbd6323b41a28e36c21af4b7f475 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:40:32.217856 27320 raft_consensus.cc:2258] T 30e8ffe78ad44cb8a1e046011e5225f2 P f034fbd6323b41a28e36c21af4b7f475 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:32.217921 27321 raft_consensus.cc:2229] T 30e8ffe78ad44cb8a1e046011e5225f2 P 17f7bd3caba1497985a5d062a28ec8b9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:40:32.218016 27321 raft_consensus.cc:2258] T 30e8ffe78ad44cb8a1e046011e5225f2 P 17f7bd3caba1497985a5d062a28ec8b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:32.218132 27320 ts_tablet_manager.cc:1905] T 30e8ffe78ad44cb8a1e046011e5225f2 P f034fbd6323b41a28e36c21af4b7f475: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:32.218304 27321 ts_tablet_manager.cc:1905] T 30e8ffe78ad44cb8a1e046011e5225f2 P 17f7bd3caba1497985a5d062a28ec8b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:32.219808 27320 ts_tablet_manager.cc:1918] T 30e8ffe78ad44cb8a1e046011e5225f2 P f034fbd6323b41a28e36c21af4b7f475: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:40:32.219870 27320 log.cc:1196] T 30e8ffe78ad44cb8a1e046011e5225f2 P f034fbd6323b41a28e36c21af4b7f475: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-0/wal/wals/30e8ffe78ad44cb8a1e046011e5225f2
I20260928 06:40:32.220067 27321 ts_tablet_manager.cc:1918] T 30e8ffe78ad44cb8a1e046011e5225f2 P 17f7bd3caba1497985a5d062a28ec8b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:40:32.220116 27320 ts_tablet_manager.cc:1939] T 30e8ffe78ad44cb8a1e046011e5225f2 P f034fbd6323b41a28e36c21af4b7f475: Deleting consensus metadata
I20260928 06:40:32.220216 27321 log.cc:1196] T 30e8ffe78ad44cb8a1e046011e5225f2 P 17f7bd3caba1497985a5d062a28ec8b9: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-2/wal/wals/30e8ffe78ad44cb8a1e046011e5225f2
I20260928 06:40:32.220412 26734 catalog_manager.cc:4841] TS f034fbd6323b41a28e36c21af4b7f475 (127.18.211.65:45105): tablet 30e8ffe78ad44cb8a1e046011e5225f2 (table test-table [id=513d4d7a2a034fdbb54b536afad4aff8]) successfully deleted
I20260928 06:40:32.220465 27319 tablet_replica.cc:330] T 30e8ffe78ad44cb8a1e046011e5225f2 P 0007b185a0014c199414c8c0607806f1: stopping tablet replica
I20260928 06:40:32.220453 27321 ts_tablet_manager.cc:1939] T 30e8ffe78ad44cb8a1e046011e5225f2 P 17f7bd3caba1497985a5d062a28ec8b9: Deleting consensus metadata
I20260928 06:40:32.220541 27319 raft_consensus.cc:2229] T 30e8ffe78ad44cb8a1e046011e5225f2 P 0007b185a0014c199414c8c0607806f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:40:32.220587 27319 raft_consensus.cc:2258] T 30e8ffe78ad44cb8a1e046011e5225f2 P 0007b185a0014c199414c8c0607806f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:32.220840 26735 catalog_manager.cc:4841] TS 17f7bd3caba1497985a5d062a28ec8b9 (127.18.211.67:35041): tablet 30e8ffe78ad44cb8a1e046011e5225f2 (table test-table [id=513d4d7a2a034fdbb54b536afad4aff8]) successfully deleted
I20260928 06:40:32.220867 27319 ts_tablet_manager.cc:1905] T 30e8ffe78ad44cb8a1e046011e5225f2 P 0007b185a0014c199414c8c0607806f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:32.222486 27319 ts_tablet_manager.cc:1918] T 30e8ffe78ad44cb8a1e046011e5225f2 P 0007b185a0014c199414c8c0607806f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:40:32.222545 27319 log.cc:1196] T 30e8ffe78ad44cb8a1e046011e5225f2 P 0007b185a0014c199414c8c0607806f1: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-1/wal/wals/30e8ffe78ad44cb8a1e046011e5225f2
I20260928 06:40:32.222743 27319 ts_tablet_manager.cc:1939] T 30e8ffe78ad44cb8a1e046011e5225f2 P 0007b185a0014c199414c8c0607806f1: Deleting consensus metadata
I20260928 06:40:32.223264 26733 catalog_manager.cc:4841] TS 0007b185a0014c199414c8c0607806f1 (127.18.211.66:36675): tablet 30e8ffe78ad44cb8a1e046011e5225f2 (table test-table [id=513d4d7a2a034fdbb54b536afad4aff8]) successfully deleted
Sep 28 06:40:32 dist-test-slave-b28r krb5kdc[26704](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 06:40:32 dist-test-slave-b28r krb5kdc[26704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:32 dist-test-slave-b28r krb5kdc[26704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 06:40:32.247138 26798 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:32.241959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36298 (local address 127.18.211.126:41149)
0928 06:40:32.242006 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:32.242008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:32.242173 (+ 165us) server_negotiation.cc:407] Connection header received
0928 06:40:32.242206 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:32.242207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:32.242218 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:32.242303 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:32.242814 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:32.243355 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:32.244074 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:32.244257 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:32.244317 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:32.247055 (+ 2738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36298: BlockingRecv error: recv got EOF from 127.0.0.1:36298 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 28 06:40:32 dist-test-slave-b28r krb5kdc[26704](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM
I20260928 06:40:32.255511 26798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:32.247980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36314 (local address 127.18.211.126:41149)
0928 06:40:32.248034 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:32.248036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:32.248184 (+ 148us) server_negotiation.cc:407] Connection header received
0928 06:40:32.248230 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:32.248231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:32.248242 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:32.248301 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:32.248784 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:32.249490 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:32.250156 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:32.250430 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:32.250502 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:32.252814 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:32.252816 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:32.252816 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:32.252824 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:32.254305 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:32.254629 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:32.254630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:32.254631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:32.254693 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:32.254913 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:32.254914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:32.254915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:32.255051 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:32.255249 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:32.255337 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 06:40:32.255414 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:40:32.256978 26749 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:40:32.257157 26749 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:40:32.258543 26886 tablet_service.cc:1467] Processing CreateTablet for tablet 92bfc4910d1f41f69e29959a38579e0c (DEFAULT_TABLE table=test-table [id=65f98907756a42ffa7732aa9e87661f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:32.258543 27026 tablet_service.cc:1467] Processing CreateTablet for tablet 92bfc4910d1f41f69e29959a38579e0c (DEFAULT_TABLE table=test-table [id=65f98907756a42ffa7732aa9e87661f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:32.258673 27026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92bfc4910d1f41f69e29959a38579e0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:32.258673 26886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92bfc4910d1f41f69e29959a38579e0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:32.259972 27261 tablet_bootstrap.cc:492] T 92bfc4910d1f41f69e29959a38579e0c P 0007b185a0014c199414c8c0607806f1: Bootstrap starting.
I20260928 06:40:32.260001 27166 tablet_service.cc:1467] Processing CreateTablet for tablet 92bfc4910d1f41f69e29959a38579e0c (DEFAULT_TABLE table=test-table [id=65f98907756a42ffa7732aa9e87661f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:32.260128 27166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92bfc4910d1f41f69e29959a38579e0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:32.260370 27261 tablet_bootstrap.cc:654] T 92bfc4910d1f41f69e29959a38579e0c P 0007b185a0014c199414c8c0607806f1: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:32.260686 27262 tablet_bootstrap.cc:492] T 92bfc4910d1f41f69e29959a38579e0c P f034fbd6323b41a28e36c21af4b7f475: Bootstrap starting.
I20260928 06:40:32.261053 27262 tablet_bootstrap.cc:654] T 92bfc4910d1f41f69e29959a38579e0c P f034fbd6323b41a28e36c21af4b7f475: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:32.261193 27260 tablet_bootstrap.cc:492] T 92bfc4910d1f41f69e29959a38579e0c P 17f7bd3caba1497985a5d062a28ec8b9: Bootstrap starting.
I20260928 06:40:32.261518 27261 tablet_bootstrap.cc:492] T 92bfc4910d1f41f69e29959a38579e0c P 0007b185a0014c199414c8c0607806f1: No bootstrap required, opened a new log
I20260928 06:40:32.261580 27261 ts_tablet_manager.cc:1397] T 92bfc4910d1f41f69e29959a38579e0c P 0007b185a0014c199414c8c0607806f1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:40:32.261677 27260 tablet_bootstrap.cc:654] T 92bfc4910d1f41f69e29959a38579e0c P 17f7bd3caba1497985a5d062a28ec8b9: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:32.261716 27261 raft_consensus.cc:348] T 92bfc4910d1f41f69e29959a38579e0c P 0007b185a0014c199414c8c0607806f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } }
I20260928 06:40:32.261787 27261 raft_consensus.cc:374] T 92bfc4910d1f41f69e29959a38579e0c P 0007b185a0014c199414c8c0607806f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:32.261803 27261 raft_consensus.cc:729] T 92bfc4910d1f41f69e29959a38579e0c P 0007b185a0014c199414c8c0607806f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0007b185a0014c199414c8c0607806f1, State: Initialized, Role: FOLLOWER
I20260928 06:40:32.261847 27261 consensus_queue.cc:260] T 92bfc4910d1f41f69e29959a38579e0c P 0007b185a0014c199414c8c0607806f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } }
I20260928 06:40:32.261924 27261 ts_tablet_manager.cc:1428] T 92bfc4910d1f41f69e29959a38579e0c P 0007b185a0014c199414c8c0607806f1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:32.262339 27260 tablet_bootstrap.cc:492] T 92bfc4910d1f41f69e29959a38579e0c P 17f7bd3caba1497985a5d062a28ec8b9: No bootstrap required, opened a new log
I20260928 06:40:32.262403 27260 ts_tablet_manager.cc:1397] T 92bfc4910d1f41f69e29959a38579e0c P 17f7bd3caba1497985a5d062a28ec8b9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260928 06:40:32.262426 27262 tablet_bootstrap.cc:492] T 92bfc4910d1f41f69e29959a38579e0c P f034fbd6323b41a28e36c21af4b7f475: No bootstrap required, opened a new log
I20260928 06:40:32.262470 27262 ts_tablet_manager.cc:1397] T 92bfc4910d1f41f69e29959a38579e0c P f034fbd6323b41a28e36c21af4b7f475: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:40:32.262538 27260 raft_consensus.cc:348] T 92bfc4910d1f41f69e29959a38579e0c P 17f7bd3caba1497985a5d062a28ec8b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } }
I20260928 06:40:32.262603 27260 raft_consensus.cc:374] T 92bfc4910d1f41f69e29959a38579e0c P 17f7bd3caba1497985a5d062a28ec8b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:32.262621 27260 raft_consensus.cc:729] T 92bfc4910d1f41f69e29959a38579e0c P 17f7bd3caba1497985a5d062a28ec8b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17f7bd3caba1497985a5d062a28ec8b9, State: Initialized, Role: FOLLOWER
I20260928 06:40:32.262609 27262 raft_consensus.cc:348] T 92bfc4910d1f41f69e29959a38579e0c P f034fbd6323b41a28e36c21af4b7f475 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } }
I20260928 06:40:32.262665 27262 raft_consensus.cc:374] T 92bfc4910d1f41f69e29959a38579e0c P f034fbd6323b41a28e36c21af4b7f475 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:32.262681 27262 raft_consensus.cc:729] T 92bfc4910d1f41f69e29959a38579e0c P f034fbd6323b41a28e36c21af4b7f475 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f034fbd6323b41a28e36c21af4b7f475, State: Initialized, Role: FOLLOWER
I20260928 06:40:32.262674 27260 consensus_queue.cc:260] T 92bfc4910d1f41f69e29959a38579e0c P 17f7bd3caba1497985a5d062a28ec8b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } }
I20260928 06:40:32.262727 27262 consensus_queue.cc:260] T 92bfc4910d1f41f69e29959a38579e0c P f034fbd6323b41a28e36c21af4b7f475 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } }
I20260928 06:40:32.262761 27260 ts_tablet_manager.cc:1428] T 92bfc4910d1f41f69e29959a38579e0c P 17f7bd3caba1497985a5d062a28ec8b9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:32.262810 27262 ts_tablet_manager.cc:1428] T 92bfc4910d1f41f69e29959a38579e0c P f034fbd6323b41a28e36c21af4b7f475: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:32.469508 27268 raft_consensus.cc:482] T 92bfc4910d1f41f69e29959a38579e0c P f034fbd6323b41a28e36c21af4b7f475 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:40:32.469612 27268 raft_consensus.cc:504] T 92bfc4910d1f41f69e29959a38579e0c P f034fbd6323b41a28e36c21af4b7f475 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } }
I20260928 06:40:32.469801 27268 leader_election.cc:288] T 92bfc4910d1f41f69e29959a38579e0c P f034fbd6323b41a28e36c21af4b7f475 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0007b185a0014c199414c8c0607806f1 (127.18.211.66:36675), 17f7bd3caba1497985a5d062a28ec8b9 (127.18.211.67:35041)
I20260928 06:40:32.470005 27046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bfc4910d1f41f69e29959a38579e0c" candidate_uuid: "f034fbd6323b41a28e36c21af4b7f475" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0007b185a0014c199414c8c0607806f1" is_pre_election: true
I20260928 06:40:32.470019 27186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bfc4910d1f41f69e29959a38579e0c" candidate_uuid: "f034fbd6323b41a28e36c21af4b7f475" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17f7bd3caba1497985a5d062a28ec8b9" is_pre_election: true
I20260928 06:40:32.470099 27046 raft_consensus.cc:2454] T 92bfc4910d1f41f69e29959a38579e0c P 0007b185a0014c199414c8c0607806f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f034fbd6323b41a28e36c21af4b7f475 in term 0.
I20260928 06:40:32.470108 27186 raft_consensus.cc:2454] T 92bfc4910d1f41f69e29959a38579e0c P 17f7bd3caba1497985a5d062a28ec8b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f034fbd6323b41a28e36c21af4b7f475 in term 0.
I20260928 06:40:32.470281 26837 leader_election.cc:302] T 92bfc4910d1f41f69e29959a38579e0c P f034fbd6323b41a28e36c21af4b7f475 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0007b185a0014c199414c8c0607806f1, f034fbd6323b41a28e36c21af4b7f475; no voters:
I20260928 06:40:32.470402 27268 raft_consensus.cc:2781] T 92bfc4910d1f41f69e29959a38579e0c P f034fbd6323b41a28e36c21af4b7f475 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:40:32.470435 27268 raft_consensus.cc:482] T 92bfc4910d1f41f69e29959a38579e0c P f034fbd6323b41a28e36c21af4b7f475 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:40:32.470450 27268 raft_consensus.cc:3037] T 92bfc4910d1f41f69e29959a38579e0c P f034fbd6323b41a28e36c21af4b7f475 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:32.471220 27268 raft_consensus.cc:504] T 92bfc4910d1f41f69e29959a38579e0c P f034fbd6323b41a28e36c21af4b7f475 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } }
I20260928 06:40:32.471339 27268 leader_election.cc:288] T 92bfc4910d1f41f69e29959a38579e0c P f034fbd6323b41a28e36c21af4b7f475 [CANDIDATE]: Term 1 election: Requested vote from peers 0007b185a0014c199414c8c0607806f1 (127.18.211.66:36675), 17f7bd3caba1497985a5d062a28ec8b9 (127.18.211.67:35041)
I20260928 06:40:32.471511 27186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bfc4910d1f41f69e29959a38579e0c" candidate_uuid: "f034fbd6323b41a28e36c21af4b7f475" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17f7bd3caba1497985a5d062a28ec8b9"
I20260928 06:40:32.471591 27186 raft_consensus.cc:3037] T 92bfc4910d1f41f69e29959a38579e0c P 17f7bd3caba1497985a5d062a28ec8b9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:32.471580 27046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92bfc4910d1f41f69e29959a38579e0c" candidate_uuid: "f034fbd6323b41a28e36c21af4b7f475" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0007b185a0014c199414c8c0607806f1"
I20260928 06:40:32.471648 27046 raft_consensus.cc:3037] T 92bfc4910d1f41f69e29959a38579e0c P 0007b185a0014c199414c8c0607806f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:32.472332 27046 raft_consensus.cc:2454] T 92bfc4910d1f41f69e29959a38579e0c P 0007b185a0014c199414c8c0607806f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f034fbd6323b41a28e36c21af4b7f475 in term 1.
I20260928 06:40:32.472405 27186 raft_consensus.cc:2454] T 92bfc4910d1f41f69e29959a38579e0c P 17f7bd3caba1497985a5d062a28ec8b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f034fbd6323b41a28e36c21af4b7f475 in term 1.
I20260928 06:40:32.472546 26837 leader_election.cc:302] T 92bfc4910d1f41f69e29959a38579e0c P f034fbd6323b41a28e36c21af4b7f475 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0007b185a0014c199414c8c0607806f1, f034fbd6323b41a28e36c21af4b7f475; no voters:
I20260928 06:40:32.472692 27268 raft_consensus.cc:2781] T 92bfc4910d1f41f69e29959a38579e0c P f034fbd6323b41a28e36c21af4b7f475 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:32.472826 27268 raft_consensus.cc:686] T 92bfc4910d1f41f69e29959a38579e0c P f034fbd6323b41a28e36c21af4b7f475 [term 1 LEADER]: Becoming Leader. State: Replica: f034fbd6323b41a28e36c21af4b7f475, State: Running, Role: LEADER
I20260928 06:40:32.472899 27268 consensus_queue.cc:237] T 92bfc4910d1f41f69e29959a38579e0c P f034fbd6323b41a28e36c21af4b7f475 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } }
I20260928 06:40:32.473613 26748 catalog_manager.cc:5495] T 92bfc4910d1f41f69e29959a38579e0c P f034fbd6323b41a28e36c21af4b7f475 reported cstate change: term changed from 0 to 1, leader changed from <none> to f034fbd6323b41a28e36c21af4b7f475 (127.18.211.65). New cstate: current_term: 1 leader_uuid: "f034fbd6323b41a28e36c21af4b7f475" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } health_report { overall_health: UNKNOWN } } }
I20260928 06:40:32.489852 27340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:32.487418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45998 (local address 127.18.211.65:45105)
0928 06:40:32.487573 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:32.487575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:32.487582 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:40:32.487619 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:32.487620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:32.487640 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:32.487708 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:40:32.487850 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:32.488406 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:32.489162 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:32.489392 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:32.489445 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:32.489547 (+ 102us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:40:32.489649 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:40:32.489695 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:32.489731 (+ 36us) server_negotiation.cc:299] Negotiation successful
0928 06:40:32.489748 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":98,"threads_started":1}
I20260928 06:40:32.490841 27046 raft_consensus.cc:1261] T 92bfc4910d1f41f69e29959a38579e0c P 0007b185a0014c199414c8c0607806f1 [term 1 FOLLOWER]: Refusing update from remote peer f034fbd6323b41a28e36c21af4b7f475: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:40:32.490885 27186 raft_consensus.cc:1261] T 92bfc4910d1f41f69e29959a38579e0c P 17f7bd3caba1497985a5d062a28ec8b9 [term 1 FOLLOWER]: Refusing update from remote peer f034fbd6323b41a28e36c21af4b7f475: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:40:32.491142 27337 consensus_queue.cc:1035] T 92bfc4910d1f41f69e29959a38579e0c P f034fbd6323b41a28e36c21af4b7f475 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:40:32.491307 27268 consensus_queue.cc:1035] T 92bfc4910d1f41f69e29959a38579e0c P f034fbd6323b41a28e36c21af4b7f475 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:40:32.494607 26749 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:40:32.494689 26749 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:40:32.495256 26749 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29696cea0aef44818a5a20643af487e8: Sending DeleteTablet for 3 replicas of tablet 92bfc4910d1f41f69e29959a38579e0c
I20260928 06:40:32.495522 27166 tablet_service.cc:1514] Processing DeleteTablet for tablet 92bfc4910d1f41f69e29959a38579e0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:32 UTC) from {username='oryx', principal='oryx/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:58852
I20260928 06:40:32.495530 26886 tablet_service.cc:1514] Processing DeleteTablet for tablet 92bfc4910d1f41f69e29959a38579e0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:32 UTC) from {username='oryx', principal='oryx/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:45992
I20260928 06:40:32.495617 27320 tablet_replica.cc:330] T 92bfc4910d1f41f69e29959a38579e0c P f034fbd6323b41a28e36c21af4b7f475: stopping tablet replica
I20260928 06:40:32.495621 27026 tablet_service.cc:1514] Processing DeleteTablet for tablet 92bfc4910d1f41f69e29959a38579e0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:32 UTC) from {username='oryx', principal='oryx/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:60706
I20260928 06:40:32.495664 27320 raft_consensus.cc:2229] T 92bfc4910d1f41f69e29959a38579e0c P f034fbd6323b41a28e36c21af4b7f475 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:40:32.495700 27321 tablet_replica.cc:330] T 92bfc4910d1f41f69e29959a38579e0c P 17f7bd3caba1497985a5d062a28ec8b9: stopping tablet replica
I20260928 06:40:32.495723 27320 raft_consensus.cc:2258] T 92bfc4910d1f41f69e29959a38579e0c P f034fbd6323b41a28e36c21af4b7f475 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:32.495761 27321 raft_consensus.cc:2229] T 92bfc4910d1f41f69e29959a38579e0c P 17f7bd3caba1497985a5d062a28ec8b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:40:32.495806 27321 raft_consensus.cc:2258] T 92bfc4910d1f41f69e29959a38579e0c P 17f7bd3caba1497985a5d062a28ec8b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:32.495882 27319 tablet_replica.cc:330] T 92bfc4910d1f41f69e29959a38579e0c P 0007b185a0014c199414c8c0607806f1: stopping tablet replica
I20260928 06:40:32.495929 27319 raft_consensus.cc:2229] T 92bfc4910d1f41f69e29959a38579e0c P 0007b185a0014c199414c8c0607806f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:40:32.495959 27319 raft_consensus.cc:2258] T 92bfc4910d1f41f69e29959a38579e0c P 0007b185a0014c199414c8c0607806f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:32.496222 27319 ts_tablet_manager.cc:1905] T 92bfc4910d1f41f69e29959a38579e0c P 0007b185a0014c199414c8c0607806f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:32.496243 27320 ts_tablet_manager.cc:1905] T 92bfc4910d1f41f69e29959a38579e0c P f034fbd6323b41a28e36c21af4b7f475: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:32.497479 27319 ts_tablet_manager.cc:1918] T 92bfc4910d1f41f69e29959a38579e0c P 0007b185a0014c199414c8c0607806f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:40:32.497529 27319 log.cc:1196] T 92bfc4910d1f41f69e29959a38579e0c P 0007b185a0014c199414c8c0607806f1: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-1/wal/wals/92bfc4910d1f41f69e29959a38579e0c
I20260928 06:40:32.497712 27319 ts_tablet_manager.cc:1939] T 92bfc4910d1f41f69e29959a38579e0c P 0007b185a0014c199414c8c0607806f1: Deleting consensus metadata
I20260928 06:40:32.497715 27320 ts_tablet_manager.cc:1918] T 92bfc4910d1f41f69e29959a38579e0c P f034fbd6323b41a28e36c21af4b7f475: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:40:32.497766 27320 log.cc:1196] T 92bfc4910d1f41f69e29959a38579e0c P f034fbd6323b41a28e36c21af4b7f475: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-0/wal/wals/92bfc4910d1f41f69e29959a38579e0c
I20260928 06:40:32.497962 27320 ts_tablet_manager.cc:1939] T 92bfc4910d1f41f69e29959a38579e0c P f034fbd6323b41a28e36c21af4b7f475: Deleting consensus metadata
I20260928 06:40:32.497992 26733 catalog_manager.cc:4841] TS 0007b185a0014c199414c8c0607806f1 (127.18.211.66:36675): tablet 92bfc4910d1f41f69e29959a38579e0c (table test-table [id=65f98907756a42ffa7732aa9e87661f8]) successfully deleted
I20260928 06:40:32.498219 26734 catalog_manager.cc:4841] TS f034fbd6323b41a28e36c21af4b7f475 (127.18.211.65:45105): tablet 92bfc4910d1f41f69e29959a38579e0c (table test-table [id=65f98907756a42ffa7732aa9e87661f8]) successfully deleted
I20260928 06:40:32.498258 26749 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:40:32.498389 26749 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:40:32.498417 27321 ts_tablet_manager.cc:1905] T 92bfc4910d1f41f69e29959a38579e0c P 17f7bd3caba1497985a5d062a28ec8b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:32.499954 27321 ts_tablet_manager.cc:1918] T 92bfc4910d1f41f69e29959a38579e0c P 17f7bd3caba1497985a5d062a28ec8b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:40:32.500013 27321 log.cc:1196] T 92bfc4910d1f41f69e29959a38579e0c P 17f7bd3caba1497985a5d062a28ec8b9: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-2/wal/wals/92bfc4910d1f41f69e29959a38579e0c
I20260928 06:40:32.500207 27321 ts_tablet_manager.cc:1939] T 92bfc4910d1f41f69e29959a38579e0c P 17f7bd3caba1497985a5d062a28ec8b9: Deleting consensus metadata
I20260928 06:40:32.500460 27026 tablet_service.cc:1467] Processing CreateTablet for tablet 5f1e16344abd41838773f4609c051b9a (DEFAULT_TABLE table=test-table [id=0d3caf19243c456fb2af16e3a3808062]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:32.500536 26735 catalog_manager.cc:4841] TS 17f7bd3caba1497985a5d062a28ec8b9 (127.18.211.67:35041): tablet 92bfc4910d1f41f69e29959a38579e0c (table test-table [id=65f98907756a42ffa7732aa9e87661f8]) successfully deleted
I20260928 06:40:32.500582 27026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f1e16344abd41838773f4609c051b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:32.501375 27261 tablet_bootstrap.cc:492] T 5f1e16344abd41838773f4609c051b9a P 0007b185a0014c199414c8c0607806f1: Bootstrap starting.
I20260928 06:40:32.501397 27166 tablet_service.cc:1467] Processing CreateTablet for tablet 5f1e16344abd41838773f4609c051b9a (DEFAULT_TABLE table=test-table [id=0d3caf19243c456fb2af16e3a3808062]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:32.501482 27166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f1e16344abd41838773f4609c051b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:32.501751 27261 tablet_bootstrap.cc:654] T 5f1e16344abd41838773f4609c051b9a P 0007b185a0014c199414c8c0607806f1: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:32.502501 27260 tablet_bootstrap.cc:492] T 5f1e16344abd41838773f4609c051b9a P 17f7bd3caba1497985a5d062a28ec8b9: Bootstrap starting.
I20260928 06:40:32.502818 27260 tablet_bootstrap.cc:654] T 5f1e16344abd41838773f4609c051b9a P 17f7bd3caba1497985a5d062a28ec8b9: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:32.503005 27261 tablet_bootstrap.cc:492] T 5f1e16344abd41838773f4609c051b9a P 0007b185a0014c199414c8c0607806f1: No bootstrap required, opened a new log
I20260928 06:40:32.503046 27261 ts_tablet_manager.cc:1397] T 5f1e16344abd41838773f4609c051b9a P 0007b185a0014c199414c8c0607806f1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:40:32.503053 26886 tablet_service.cc:1467] Processing CreateTablet for tablet 5f1e16344abd41838773f4609c051b9a (DEFAULT_TABLE table=test-table [id=0d3caf19243c456fb2af16e3a3808062]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:32.503166 26886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f1e16344abd41838773f4609c051b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:32.503202 27261 raft_consensus.cc:348] T 5f1e16344abd41838773f4609c051b9a P 0007b185a0014c199414c8c0607806f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } }
I20260928 06:40:32.503264 27261 raft_consensus.cc:374] T 5f1e16344abd41838773f4609c051b9a P 0007b185a0014c199414c8c0607806f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:32.503276 27261 raft_consensus.cc:729] T 5f1e16344abd41838773f4609c051b9a P 0007b185a0014c199414c8c0607806f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0007b185a0014c199414c8c0607806f1, State: Initialized, Role: FOLLOWER
I20260928 06:40:32.503312 27261 consensus_queue.cc:260] T 5f1e16344abd41838773f4609c051b9a P 0007b185a0014c199414c8c0607806f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } }
I20260928 06:40:32.503405 27261 ts_tablet_manager.cc:1428] T 5f1e16344abd41838773f4609c051b9a P 0007b185a0014c199414c8c0607806f1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:32.504257 27262 tablet_bootstrap.cc:492] T 5f1e16344abd41838773f4609c051b9a P f034fbd6323b41a28e36c21af4b7f475: Bootstrap starting.
I20260928 06:40:32.504623 27260 tablet_bootstrap.cc:492] T 5f1e16344abd41838773f4609c051b9a P 17f7bd3caba1497985a5d062a28ec8b9: No bootstrap required, opened a new log
I20260928 06:40:32.504648 27262 tablet_bootstrap.cc:654] T 5f1e16344abd41838773f4609c051b9a P f034fbd6323b41a28e36c21af4b7f475: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:32.504664 27260 ts_tablet_manager.cc:1397] T 5f1e16344abd41838773f4609c051b9a P 17f7bd3caba1497985a5d062a28ec8b9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260928 06:40:32.504757 27260 raft_consensus.cc:348] T 5f1e16344abd41838773f4609c051b9a P 17f7bd3caba1497985a5d062a28ec8b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } }
I20260928 06:40:32.504822 27260 raft_consensus.cc:374] T 5f1e16344abd41838773f4609c051b9a P 17f7bd3caba1497985a5d062a28ec8b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:32.504842 27260 raft_consensus.cc:729] T 5f1e16344abd41838773f4609c051b9a P 17f7bd3caba1497985a5d062a28ec8b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17f7bd3caba1497985a5d062a28ec8b9, State: Initialized, Role: FOLLOWER
I20260928 06:40:32.504889 27260 consensus_queue.cc:260] T 5f1e16344abd41838773f4609c051b9a P 17f7bd3caba1497985a5d062a28ec8b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } }
I20260928 06:40:32.504969 27260 ts_tablet_manager.cc:1428] T 5f1e16344abd41838773f4609c051b9a P 17f7bd3caba1497985a5d062a28ec8b9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:32.505255 27262 tablet_bootstrap.cc:492] T 5f1e16344abd41838773f4609c051b9a P f034fbd6323b41a28e36c21af4b7f475: No bootstrap required, opened a new log
I20260928 06:40:32.505294 27262 ts_tablet_manager.cc:1397] T 5f1e16344abd41838773f4609c051b9a P f034fbd6323b41a28e36c21af4b7f475: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:32.505424 27262 raft_consensus.cc:348] T 5f1e16344abd41838773f4609c051b9a P f034fbd6323b41a28e36c21af4b7f475 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } }
I20260928 06:40:32.505496 27262 raft_consensus.cc:374] T 5f1e16344abd41838773f4609c051b9a P f034fbd6323b41a28e36c21af4b7f475 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:32.505508 27262 raft_consensus.cc:729] T 5f1e16344abd41838773f4609c051b9a P f034fbd6323b41a28e36c21af4b7f475 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f034fbd6323b41a28e36c21af4b7f475, State: Initialized, Role: FOLLOWER
I20260928 06:40:32.505534 27262 consensus_queue.cc:260] T 5f1e16344abd41838773f4609c051b9a P f034fbd6323b41a28e36c21af4b7f475 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } }
I20260928 06:40:32.505599 27262 ts_tablet_manager.cc:1428] T 5f1e16344abd41838773f4609c051b9a P f034fbd6323b41a28e36c21af4b7f475: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:32.522145 27266 raft_consensus.cc:482] T 5f1e16344abd41838773f4609c051b9a P 17f7bd3caba1497985a5d062a28ec8b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:40:32.522233 27266 raft_consensus.cc:504] T 5f1e16344abd41838773f4609c051b9a P 17f7bd3caba1497985a5d062a28ec8b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } }
I20260928 06:40:32.522361 27266 leader_election.cc:288] T 5f1e16344abd41838773f4609c051b9a P 17f7bd3caba1497985a5d062a28ec8b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f034fbd6323b41a28e36c21af4b7f475 (127.18.211.65:45105), 0007b185a0014c199414c8c0607806f1 (127.18.211.66:36675)
I20260928 06:40:32.522558 26906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f1e16344abd41838773f4609c051b9a" candidate_uuid: "17f7bd3caba1497985a5d062a28ec8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f034fbd6323b41a28e36c21af4b7f475" is_pre_election: true
I20260928 06:40:32.522605 27046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f1e16344abd41838773f4609c051b9a" candidate_uuid: "17f7bd3caba1497985a5d062a28ec8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0007b185a0014c199414c8c0607806f1" is_pre_election: true
I20260928 06:40:32.522646 26906 raft_consensus.cc:2454] T 5f1e16344abd41838773f4609c051b9a P f034fbd6323b41a28e36c21af4b7f475 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17f7bd3caba1497985a5d062a28ec8b9 in term 0.
I20260928 06:40:32.522676 27046 raft_consensus.cc:2454] T 5f1e16344abd41838773f4609c051b9a P 0007b185a0014c199414c8c0607806f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17f7bd3caba1497985a5d062a28ec8b9 in term 0.
I20260928 06:40:32.522810 27118 leader_election.cc:302] T 5f1e16344abd41838773f4609c051b9a P 17f7bd3caba1497985a5d062a28ec8b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17f7bd3caba1497985a5d062a28ec8b9, f034fbd6323b41a28e36c21af4b7f475; no voters:
I20260928 06:40:32.522933 27266 raft_consensus.cc:2781] T 5f1e16344abd41838773f4609c051b9a P 17f7bd3caba1497985a5d062a28ec8b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:40:32.522964 27266 raft_consensus.cc:482] T 5f1e16344abd41838773f4609c051b9a P 17f7bd3caba1497985a5d062a28ec8b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:40:32.522977 27266 raft_consensus.cc:3037] T 5f1e16344abd41838773f4609c051b9a P 17f7bd3caba1497985a5d062a28ec8b9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:32.523721 27266 raft_consensus.cc:504] T 5f1e16344abd41838773f4609c051b9a P 17f7bd3caba1497985a5d062a28ec8b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } }
I20260928 06:40:32.523860 27266 leader_election.cc:288] T 5f1e16344abd41838773f4609c051b9a P 17f7bd3caba1497985a5d062a28ec8b9 [CANDIDATE]: Term 1 election: Requested vote from peers f034fbd6323b41a28e36c21af4b7f475 (127.18.211.65:45105), 0007b185a0014c199414c8c0607806f1 (127.18.211.66:36675)
I20260928 06:40:32.524003 26906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f1e16344abd41838773f4609c051b9a" candidate_uuid: "17f7bd3caba1497985a5d062a28ec8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f034fbd6323b41a28e36c21af4b7f475"
I20260928 06:40:32.524042 27046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f1e16344abd41838773f4609c051b9a" candidate_uuid: "17f7bd3caba1497985a5d062a28ec8b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0007b185a0014c199414c8c0607806f1"
I20260928 06:40:32.524081 26906 raft_consensus.cc:3037] T 5f1e16344abd41838773f4609c051b9a P f034fbd6323b41a28e36c21af4b7f475 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:32.524111 27046 raft_consensus.cc:3037] T 5f1e16344abd41838773f4609c051b9a P 0007b185a0014c199414c8c0607806f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:32.524667 27046 raft_consensus.cc:2454] T 5f1e16344abd41838773f4609c051b9a P 0007b185a0014c199414c8c0607806f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17f7bd3caba1497985a5d062a28ec8b9 in term 1.
I20260928 06:40:32.524813 26906 raft_consensus.cc:2454] T 5f1e16344abd41838773f4609c051b9a P f034fbd6323b41a28e36c21af4b7f475 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17f7bd3caba1497985a5d062a28ec8b9 in term 1.
I20260928 06:40:32.524833 27117 leader_election.cc:302] T 5f1e16344abd41838773f4609c051b9a P 17f7bd3caba1497985a5d062a28ec8b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0007b185a0014c199414c8c0607806f1, 17f7bd3caba1497985a5d062a28ec8b9; no voters:
I20260928 06:40:32.524911 27266 raft_consensus.cc:2781] T 5f1e16344abd41838773f4609c051b9a P 17f7bd3caba1497985a5d062a28ec8b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:32.524953 27266 raft_consensus.cc:686] T 5f1e16344abd41838773f4609c051b9a P 17f7bd3caba1497985a5d062a28ec8b9 [term 1 LEADER]: Becoming Leader. State: Replica: 17f7bd3caba1497985a5d062a28ec8b9, State: Running, Role: LEADER
I20260928 06:40:32.524991 27266 consensus_queue.cc:237] T 5f1e16344abd41838773f4609c051b9a P 17f7bd3caba1497985a5d062a28ec8b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } }
I20260928 06:40:32.525569 26749 catalog_manager.cc:5495] T 5f1e16344abd41838773f4609c051b9a P 17f7bd3caba1497985a5d062a28ec8b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17f7bd3caba1497985a5d062a28ec8b9 (127.18.211.67). New cstate: current_term: 1 leader_uuid: "17f7bd3caba1497985a5d062a28ec8b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17f7bd3caba1497985a5d062a28ec8b9" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 35041 } health_report { overall_health: HEALTHY } } }
I20260928 06:40:32.534045 27258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:32.532047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58886 (local address 127.18.211.67:35041)
0928 06:40:32.532095 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:32.532096 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:32.532103 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:40:32.532123 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:32.532123 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:32.532133 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:32.532176 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:40:32.532292 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:32.532761 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:32.533444 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:32.533644 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:32.533697 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:32.533825 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:40:32.533859 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:40:32.533900 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:32.533958 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 06:40:32.533975 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:40:32.535988 27046 raft_consensus.cc:1261] T 5f1e16344abd41838773f4609c051b9a P 0007b185a0014c199414c8c0607806f1 [term 1 FOLLOWER]: Refusing update from remote peer 17f7bd3caba1497985a5d062a28ec8b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:40:32.536199 27315 consensus_queue.cc:1035] T 5f1e16344abd41838773f4609c051b9a P 17f7bd3caba1497985a5d062a28ec8b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0007b185a0014c199414c8c0607806f1" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:40:32.536396 26906 raft_consensus.cc:1261] T 5f1e16344abd41838773f4609c051b9a P f034fbd6323b41a28e36c21af4b7f475 [term 1 FOLLOWER]: Refusing update from remote peer 17f7bd3caba1497985a5d062a28ec8b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:40:32.536831 27270 consensus_queue.cc:1035] T 5f1e16344abd41838773f4609c051b9a P 17f7bd3caba1497985a5d062a28ec8b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f034fbd6323b41a28e36c21af4b7f475" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:40:32.546525 26748 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:40:32.546618 26748 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:40:32.547188 26748 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29696cea0aef44818a5a20643af487e8: Sending DeleteTablet for 3 replicas of tablet 5f1e16344abd41838773f4609c051b9a
I20260928 06:40:32.547552 27026 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f1e16344abd41838773f4609c051b9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:32 UTC) from {username='oryx', principal='oryx/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:60706
I20260928 06:40:32.547648 27319 tablet_replica.cc:330] T 5f1e16344abd41838773f4609c051b9a P 0007b185a0014c199414c8c0607806f1: stopping tablet replica
I20260928 06:40:32.547701 27319 raft_consensus.cc:2229] T 5f1e16344abd41838773f4609c051b9a P 0007b185a0014c199414c8c0607806f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:40:32.547734 27319 raft_consensus.cc:2258] T 5f1e16344abd41838773f4609c051b9a P 0007b185a0014c199414c8c0607806f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:32.547708 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 26821
I20260928 06:40:32.548640 27166 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f1e16344abd41838773f4609c051b9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:32 UTC) from {username='oryx', principal='oryx/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:58852
I20260928 06:40:32.548871 27319 ts_tablet_manager.cc:1905] T 5f1e16344abd41838773f4609c051b9a P 0007b185a0014c199414c8c0607806f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:32.548979 27321 tablet_replica.cc:330] T 5f1e16344abd41838773f4609c051b9a P 17f7bd3caba1497985a5d062a28ec8b9: stopping tablet replica
I20260928 06:40:32.549041 27321 raft_consensus.cc:2229] T 5f1e16344abd41838773f4609c051b9a P 17f7bd3caba1497985a5d062a28ec8b9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:40:32.549103 27321 raft_consensus.cc:2258] T 5f1e16344abd41838773f4609c051b9a P 17f7bd3caba1497985a5d062a28ec8b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:32.549372 27321 ts_tablet_manager.cc:1905] T 5f1e16344abd41838773f4609c051b9a P 17f7bd3caba1497985a5d062a28ec8b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:32.550415 27319 ts_tablet_manager.cc:1918] T 5f1e16344abd41838773f4609c051b9a P 0007b185a0014c199414c8c0607806f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:40:32.550462 27319 log.cc:1196] T 5f1e16344abd41838773f4609c051b9a P 0007b185a0014c199414c8c0607806f1: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-1/wal/wals/5f1e16344abd41838773f4609c051b9a
I20260928 06:40:32.550634 27319 ts_tablet_manager.cc:1939] T 5f1e16344abd41838773f4609c051b9a P 0007b185a0014c199414c8c0607806f1: Deleting consensus metadata
I20260928 06:40:32.550762 27321 ts_tablet_manager.cc:1918] T 5f1e16344abd41838773f4609c051b9a P 17f7bd3caba1497985a5d062a28ec8b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 06:40:32.550822 27321 log.cc:1196] T 5f1e16344abd41838773f4609c051b9a P 17f7bd3caba1497985a5d062a28ec8b9: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790577602522428-19277-0/minicluster-data/ts-2/wal/wals/5f1e16344abd41838773f4609c051b9a
I20260928 06:40:32.550959 26733 catalog_manager.cc:4841] TS 0007b185a0014c199414c8c0607806f1 (127.18.211.66:36675): tablet 5f1e16344abd41838773f4609c051b9a (table test-table [id=0d3caf19243c456fb2af16e3a3808062]) successfully deleted
I20260928 06:40:32.550995 27321 ts_tablet_manager.cc:1939] T 5f1e16344abd41838773f4609c051b9a P 17f7bd3caba1497985a5d062a28ec8b9: Deleting consensus metadata
I20260928 06:40:32.551285 26735 catalog_manager.cc:4841] TS 17f7bd3caba1497985a5d062a28ec8b9 (127.18.211.67:35041): tablet 5f1e16344abd41838773f4609c051b9a (table test-table [id=0d3caf19243c456fb2af16e3a3808062]) successfully deleted
W20260928 06:40:32.553579 26734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.211.65:45105 (error 108)
I20260928 06:40:32.553799 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 26961
I20260928 06:40:32.554563 27352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:32.553900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:45105 (local address 127.0.0.1:46010)
0928 06:40:32.554476 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:32.554523 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.65:45105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":34,"threads_started":1}
W20260928 06:40:32.554693 26734 catalog_manager.cc:4584] TS f034fbd6323b41a28e36c21af4b7f475 (127.18.211.65:45105): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5f1e16344abd41838773f4609c051b9a: Network error: Client connection negotiation failed: client connection to 127.18.211.65:45105: connect: Connection refused (error 111)
I20260928 06:40:32.558126 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 27101
I20260928 06:40:32.563040 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 26717
2026-09-28T06:40:32Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2662 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:40:32 dist-test-slave-b28r krb5kdc[27355](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:40:32 dist-test-slave-b28r krb5kdc[27355](info): set up 2 sockets
Sep 28 06:40:32 dist-test-slave-b28r krb5kdc[27355](info): commencing operation
krb5kdc: starting...
W20260928 06:40:33.516537 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:40:33 dist-test-slave-b28r krb5kdc[27355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577633, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:40:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:40:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.211.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
I20260928 06:40:33.624276 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:33975
--webserver_interface=127.18.211.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:35609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:33975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:33.703487 27368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:33.703738 27368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:33.703759 27368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:33.704910 27368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:40:33.704952 27368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:33.704965 27368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:33.704977 27368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:40:33.704988 27368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:40:33.706437 27368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:35609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:33975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:33975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:33.706660 27368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:33.706871 27368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:33.711506 27373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:33.711519 27376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:33.711555 27374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:33.711652 27368 server_base.cc:1034] running on GCE node
I20260928 06:40:33.711951 27368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:33.712255 27368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:33.713392 27368 hybrid_clock.cc:648] HybridClock initialized: now 1790577633713382 us; error 31 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T06:40:33Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1159 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:40:33 dist-test-slave-b28r krb5kdc[27381](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:40:33 dist-test-slave-b28r krb5kdc[27381](info): set up 2 sockets
Sep 28 06:40:33 dist-test-slave-b28r krb5kdc[27381](info): commencing operation
krb5kdc: starting...
W20260928 06:40:34.673111 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:40:34 dist-test-slave-b28r krb5kdc[27381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:40:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:40:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.211.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
I20260928 06:40:34.781863 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:34589
--webserver_interface=127.18.211.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:42391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:34589
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:34.859906 27394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:34.860100 27394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:34.860121 27394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:34.861229 27394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:40:34.861271 27394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:34.861284 27394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:34.861294 27394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:40:34.861312 27394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:40:34.862722 27394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:42391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:34589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:34589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:34.862933 27394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:34.863157 27394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:34.867683 27402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:34.867684 27400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:34.867720 27399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:34.868000 27394 server_base.cc:1034] running on GCE node
I20260928 06:40:34.868237 27394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:34.868534 27394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:34.869674 27394 hybrid_clock.cc:648] HybridClock initialized: now 1790577634869660 us; error 39 us; skew 500 ppm
Sep 28 06:40:34 dist-test-slave-b28r krb5kdc[27381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577634, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:34.872068 27394 init.cc:377] Logged in from keytab as kudu/127.18.211.126@KRBTEST.COM (short username kudu)
I20260928 06:40:34.872475 27394 webserver.cc:460] Webserver started at http://127.18.211.126:41651/ using document root <none> and password file <none>
I20260928 06:40:34.872665 27394 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:34.872714 27394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:34.872836 27394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:34.873844 27394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "2cd32525fadf45f8a212208cc8f98124"
format_stamp: "Formatted at 2026-09-28 06:40:34 on dist-test-slave-b28r"
server_key: "02629e0659cce1fda540c5d39958744c"
server_key_iv: "d96fb46970b130bcecb9845bb50d9596"
server_key_version: "encryptionkey@0"
I20260928 06:40:34.874179 27394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "2cd32525fadf45f8a212208cc8f98124"
format_stamp: "Formatted at 2026-09-28 06:40:34 on dist-test-slave-b28r"
server_key: "02629e0659cce1fda540c5d39958744c"
server_key_iv: "d96fb46970b130bcecb9845bb50d9596"
server_key_version: "encryptionkey@0"
I20260928 06:40:34.875428 27394 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:40:34.876071 27409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:34.876214 27394 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:40:34.876293 27394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "2cd32525fadf45f8a212208cc8f98124"
format_stamp: "Formatted at 2026-09-28 06:40:34 on dist-test-slave-b28r"
server_key: "02629e0659cce1fda540c5d39958744c"
server_key_iv: "d96fb46970b130bcecb9845bb50d9596"
server_key_version: "encryptionkey@0"
I20260928 06:40:34.876353 27394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:34.893348 27394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:34.895409 27394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:34.895537 27394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:34.899117 27394 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:34589
I20260928 06:40:34.899155 27461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:34589 every 8 connection(s)
I20260928 06:40:34.899480 27394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:40:34.900318 27462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:34.902566 27462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124: Bootstrap starting.
I20260928 06:40:34.903144 27462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:34.903405 27462 log.cc:824] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:34.904104 27462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124: No bootstrap required, opened a new log
I20260928 06:40:34.905767 27462 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd32525fadf45f8a212208cc8f98124" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34589 } }
I20260928 06:40:34.905929 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 27394
I20260928 06:40:34.905953 27462 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:34.905989 27462 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cd32525fadf45f8a212208cc8f98124, State: Initialized, Role: FOLLOWER
I20260928 06:40:34.906035 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:40:34.906157 27462 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd32525fadf45f8a212208cc8f98124" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34589 } }
I20260928 06:40:34.906230 19277 external_mini_cluster.cc:1428] Setting key 2848b42c73e6cbd78f6aeff9b3725e66
I20260928 06:40:34.906244 27462 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:40:34.906286 27462 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:40:34.906340 27462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:34.907352 27462 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd32525fadf45f8a212208cc8f98124" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34589 } }
I20260928 06:40:34.907474 27462 leader_election.cc:302] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cd32525fadf45f8a212208cc8f98124; no voters:
I20260928 06:40:34.907670 27462 leader_election.cc:288] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:40:34.907703 27467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:34.907859 27467 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124 [term 1 LEADER]: Becoming Leader. State: Replica: 2cd32525fadf45f8a212208cc8f98124, State: Running, Role: LEADER
I20260928 06:40:34.907958 27462 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:40:34.908010 27467 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd32525fadf45f8a212208cc8f98124" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34589 } }
I20260928 06:40:34.908346 27467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cd32525fadf45f8a212208cc8f98124. Latest consensus state: current_term: 1 leader_uuid: "2cd32525fadf45f8a212208cc8f98124" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd32525fadf45f8a212208cc8f98124" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34589 } } }
I20260928 06:40:34.908612 27467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124 [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:34.908980 27476 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:40:34.909490 27476 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:40:34.908540 27468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2cd32525fadf45f8a212208cc8f98124" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd32525fadf45f8a212208cc8f98124" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34589 } } }
I20260928 06:40:34.909934 27468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124 [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:34.911162 27476 catalog_manager.cc:1347] Generated new cluster ID: 003909d79cd74a4d83cc93be4aaa097b
I20260928 06:40:34.911212 27476 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:40:34 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:34.916792 27466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:34.906690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59506 (local address 127.18.211.126:34589)
0928 06:40:34.906954 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:34.906962 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:34.906986 (+ 24us) server_negotiation.cc:407] Connection header received
0928 06:40:34.907077 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:34.907109 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:34.907163 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:34.907272 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:34.908157 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:34.909558 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:34.910599 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:34.911120 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:34.911234 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:34.913537 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:34.913542 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:34.913542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:34.913555 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:34.915280 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:34.915613 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:34.915616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:34.915616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:34.915719 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:34.915911 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:34.915911 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:34.915912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:34.916164 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:34.916304 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:34.916439 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 06:40:34.916575 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":164,"threads_started":1}
I20260928 06:40:34.916841 27476 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:40:34.917829 27476 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:40:34.921823 27476 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124: Generated new TSK 0
I20260928 06:40:34.921960 27476 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.211.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab.
Entry for principal kudu/127.18.211.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab.
I20260928 06:40:34.947269 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.65:0
--local_ip_for_outbound_sockets=127.18.211.65
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:34589
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:42391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:35.030351 27488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:35.030558 27488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:35.030580 27488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:35.031798 27488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:35.031838 27488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:35.031864 27488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.65
I20260928 06:40:35.033270 27488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:42391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:34589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.65
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:35.033484 27488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:35.033697 27488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:35.038409 27493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:35.038468 27488 server_base.cc:1034] running on GCE node
W20260928 06:40:35.038434 27496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:35.038587 27494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:35.038859 27488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:35.039124 27488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:35.040295 27488 hybrid_clock.cc:648] HybridClock initialized: now 1790577635040285 us; error 26 us; skew 500 ppm
Sep 28 06:40:35 dist-test-slave-b28r krb5kdc[27381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577635, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:35.042757 27488 init.cc:377] Logged in from keytab as kudu/127.18.211.65@KRBTEST.COM (short username kudu)
I20260928 06:40:35.043242 27488 webserver.cc:460] Webserver started at http://127.18.211.65:34063/ using document root <none> and password file <none>
I20260928 06:40:35.043466 27488 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:35.043509 27488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:35.043624 27488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:35.044793 27488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-0/data/instance:
uuid: "9e5b953aba534625adbab186dd9fab34"
format_stamp: "Formatted at 2026-09-28 06:40:35 on dist-test-slave-b28r"
server_key: "e1fa5ac77ddd5e10e00c29a61c6efa9a"
server_key_iv: "d517cb45efaadba7c1841a92ff465348"
server_key_version: "encryptionkey@0"
I20260928 06:40:35.045122 27488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance:
uuid: "9e5b953aba534625adbab186dd9fab34"
format_stamp: "Formatted at 2026-09-28 06:40:35 on dist-test-slave-b28r"
server_key: "e1fa5ac77ddd5e10e00c29a61c6efa9a"
server_key_iv: "d517cb45efaadba7c1841a92ff465348"
server_key_version: "encryptionkey@0"
I20260928 06:40:35.046356 27488 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:40:35.047017 27503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:35.047174 27488 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:40:35.047294 27488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-0/wal
uuid: "9e5b953aba534625adbab186dd9fab34"
format_stamp: "Formatted at 2026-09-28 06:40:35 on dist-test-slave-b28r"
server_key: "e1fa5ac77ddd5e10e00c29a61c6efa9a"
server_key_iv: "d517cb45efaadba7c1841a92ff465348"
server_key_version: "encryptionkey@0"
I20260928 06:40:35.047360 27488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:35.062844 27488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:35.064956 27488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:35.065084 27488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:35.065310 27488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:35.065598 27488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:35.065636 27488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:35.065656 27488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:35.065680 27488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:35.071267 27488 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.65:41749
I20260928 06:40:35.071322 27616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.65:41749 every 8 connection(s)
I20260928 06:40:35.071638 27488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
I20260928 06:40:35.072022 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 27488
I20260928 06:40:35.072121 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance
I20260928 06:40:35.072316 19277 external_mini_cluster.cc:1428] Setting key cbd070ed57f7743aca26038c3644d0b0
Sep 28 06:40:35 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577635, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.211.66@KRBTEST.COM; defaulting to no policy
I20260928 06:40:35.081643 27466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:35.072592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:43523 (local address 127.18.211.126:34589)
0928 06:40:35.072658 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:35.072660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:35.072670 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:40:35.073202 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:35.073203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:35.073215 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:35.073251 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:35.074439 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.075201 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.076182 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.076393 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.076448 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:35.078761 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:35.078762 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:35.078763 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:35.078771 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:35.080180 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:35.080537 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:35.080538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:35.080538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:35.080590 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:35.080884 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:35.080885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:35.080885 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:35.080991 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:35.081042 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:35.081457 (+ 415us) server_negotiation.cc:299] Negotiation successful
0928 06:40:35.081554 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:40:35.081717 27619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:35.072356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.65:43523)
0928 06:40:35.072498 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:35.072539 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:35.073109 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:35.073343 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:35.073347 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:35.074053 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:35.074337 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:35.074344 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.075307 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.075308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:35.076095 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:35.076097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.076462 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.076463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:35.076591 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:35.077069 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:35.077080 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:35.078658 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:35.080255 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:35.080261 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:35.080263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:35.080472 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:35.080673 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:35.080674 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:35.080675 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:35.080812 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:35.081096 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:35.081104 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:35.081287 (+ 183us) client_negotiation.cc:769] Sending connection context
0928 06:40:35.081387 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 06:40:35.081542 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":73,"threads_started":1}
I20260928 06:40:35.082289 27617 heartbeater.cc:347] Connected to a master server at 127.18.211.126:34589
I20260928 06:40:35.082410 27617 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:35.082597 27617 heartbeater.cc:511] Master 127.18.211.126:34589 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.66@KRBTEST.COM" created.
I20260928 06:40:35.083235 27426 ts_manager.cc:194] Registered new tserver with Master: 9e5b953aba534625adbab186dd9fab34 (127.18.211.65:41749)
I20260928 06:40:35.083864 27426 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.65@KRBTEST.COM'} at 127.18.211.65:43523
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab.
Entry for principal kudu/127.18.211.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab.
I20260928 06:40:35.095808 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.66:0
--local_ip_for_outbound_sockets=127.18.211.66
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:34589
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:42391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:35.175546 27622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:35.175750 27622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:35.175781 27622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:35.176921 27622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:35.176959 27622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:35.176993 27622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.66
I20260928 06:40:35.178414 27622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:42391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:34589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.66
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:35.178630 27622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:35.178848 27622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:35.183720 27627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:35.183697 27628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:35.183868 27622 server_base.cc:1034] running on GCE node
W20260928 06:40:35.183710 27630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:35.184151 27622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:35.184444 27622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:35.185621 27622 hybrid_clock.cc:648] HybridClock initialized: now 1790577635185594 us; error 37 us; skew 500 ppm
Sep 28 06:40:35 dist-test-slave-b28r krb5kdc[27381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577635, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:35.188349 27622 init.cc:377] Logged in from keytab as kudu/127.18.211.66@KRBTEST.COM (short username kudu)
I20260928 06:40:35.188793 27622 webserver.cc:460] Webserver started at http://127.18.211.66:43801/ using document root <none> and password file <none>
I20260928 06:40:35.189024 27622 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:35.189077 27622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:35.189214 27622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:35.190339 27622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-1/data/instance:
uuid: "e4257cb77c944960bbb6018e968f65c7"
format_stamp: "Formatted at 2026-09-28 06:40:35 on dist-test-slave-b28r"
server_key: "37e9f707b95a0b1a7f8dc120bc07ff16"
server_key_iv: "3bee3e9b6d547cd8cec51167112d13b5"
server_key_version: "encryptionkey@0"
I20260928 06:40:35.190723 27622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance:
uuid: "e4257cb77c944960bbb6018e968f65c7"
format_stamp: "Formatted at 2026-09-28 06:40:35 on dist-test-slave-b28r"
server_key: "37e9f707b95a0b1a7f8dc120bc07ff16"
server_key_iv: "3bee3e9b6d547cd8cec51167112d13b5"
server_key_version: "encryptionkey@0"
I20260928 06:40:35.192242 27622 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:40:35.192983 27637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:35.193166 27622 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:40:35.193260 27622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-1/wal
uuid: "e4257cb77c944960bbb6018e968f65c7"
format_stamp: "Formatted at 2026-09-28 06:40:35 on dist-test-slave-b28r"
server_key: "37e9f707b95a0b1a7f8dc120bc07ff16"
server_key_iv: "3bee3e9b6d547cd8cec51167112d13b5"
server_key_version: "encryptionkey@0"
I20260928 06:40:35.193324 27622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:35.201225 27622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:35.203413 27622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:35.203552 27622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:35.203760 27622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:35.204048 27622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:35.204085 27622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:35.204116 27622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:35.204154 27622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:35.210247 27622 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.66:41497
I20260928 06:40:35.210304 27750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.66:41497 every 8 connection(s)
I20260928 06:40:35.210608 27622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:40:35 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577635, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:35.219696 27466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:35.211320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:42399 (local address 127.18.211.126:34589)
0928 06:40:35.211373 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:35.211376 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:35.211615 (+ 239us) server_negotiation.cc:407] Connection header received
0928 06:40:35.212082 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:35.212083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:35.212092 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:35.212143 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:35.213104 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.213651 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.214661 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.214863 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.214920 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:35.217059 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:35.217061 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:35.217061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:35.217068 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:35.218478 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:35.218868 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:35.218869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:35.218869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:35.218924 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:35.219158 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:35.219159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:35.219159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:35.219277 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:35.219338 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:35.219560 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 06:40:35.219630 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:40:35.219874 27753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:35.211300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.66:42399)
0928 06:40:35.211497 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:35.211533 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:35.212000 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:35.212198 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:35.212201 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:35.212818 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:35.213006 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:35.213011 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.213752 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.213753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:35.214577 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:35.214579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.214943 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.214944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:35.215050 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:35.215629 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:35.215638 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:35.216954 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:35.218555 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:35.218561 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:35.218563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:35.218797 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:35.218981 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:35.218982 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:35.218982 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:35.219075 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:35.219339 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:35.219343 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:35.219455 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 06:40:35.219538 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 06:40:35.219654 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":94,"threads_started":1}
I20260928 06:40:35.220274 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 27622
I20260928 06:40:35.220341 27751 heartbeater.cc:347] Connected to a master server at 127.18.211.126:34589
I20260928 06:40:35.220371 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance
I20260928 06:40:35.220451 27751 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:35.220573 19277 external_mini_cluster.cc:1428] Setting key 1dc3dd2d9370213055a7eb0a962dd53c
I20260928 06:40:35.220639 27751 heartbeater.cc:511] Master 127.18.211.126:34589 requested a full tablet report, sending...
I20260928 06:40:35.221016 27425 ts_manager.cc:194] Registered new tserver with Master: e4257cb77c944960bbb6018e968f65c7 (127.18.211.66:41497)
I20260928 06:40:35.221477 27425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.66@KRBTEST.COM'} at 127.18.211.66:42399
WARNING: no policy specified for kudu/127.18.211.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab.
Entry for principal kudu/127.18.211.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab.
I20260928 06:40:35.243970 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.67:0
--local_ip_for_outbound_sockets=127.18.211.67
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:34589
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:42391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:35.321759 27756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:35.321950 27756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:35.321971 27756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:35.323146 27756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:35.323186 27756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:35.323212 27756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.67
I20260928 06:40:35.324569 27756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:42391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:34589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.67
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:35.324736 27756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:35.324903 27756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:35.329514 27761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:35.329512 27762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:35.329632 27756 server_base.cc:1034] running on GCE node
W20260928 06:40:35.329540 27764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:35.329982 27756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:35.330206 27756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:35.331339 27756 hybrid_clock.cc:648] HybridClock initialized: now 1790577635331326 us; error 39 us; skew 500 ppm
Sep 28 06:40:35 dist-test-slave-b28r krb5kdc[27381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577635, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:35.333689 27756 init.cc:377] Logged in from keytab as kudu/127.18.211.67@KRBTEST.COM (short username kudu)
I20260928 06:40:35.334118 27756 webserver.cc:460] Webserver started at http://127.18.211.67:45635/ using document root <none> and password file <none>
I20260928 06:40:35.334326 27756 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:35.334374 27756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:35.334497 27756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:35.335525 27756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-2/data/instance:
uuid: "acbdd45f4d824decb21bbd60d0e5faca"
format_stamp: "Formatted at 2026-09-28 06:40:35 on dist-test-slave-b28r"
server_key: "b464c88fb2b8c9500a1f6f6197d8a099"
server_key_iv: "22e3c3fb288d7e29ff5a125eebbf7d80"
server_key_version: "encryptionkey@0"
I20260928 06:40:35.335834 27756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance:
uuid: "acbdd45f4d824decb21bbd60d0e5faca"
format_stamp: "Formatted at 2026-09-28 06:40:35 on dist-test-slave-b28r"
server_key: "b464c88fb2b8c9500a1f6f6197d8a099"
server_key_iv: "22e3c3fb288d7e29ff5a125eebbf7d80"
server_key_version: "encryptionkey@0"
I20260928 06:40:35.337028 27756 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:40:35.337811 27771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:35.338028 27756 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:35.338107 27756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-2/wal
uuid: "acbdd45f4d824decb21bbd60d0e5faca"
format_stamp: "Formatted at 2026-09-28 06:40:35 on dist-test-slave-b28r"
server_key: "b464c88fb2b8c9500a1f6f6197d8a099"
server_key_iv: "22e3c3fb288d7e29ff5a125eebbf7d80"
server_key_version: "encryptionkey@0"
I20260928 06:40:35.338171 27756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:35.353076 27756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:35.355170 27756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:35.355305 27756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:35.355521 27756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:35.355820 27756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:35.355852 27756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:35.355880 27756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:35.355903 27756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:35.361303 27756 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.67:41863
I20260928 06:40:35.361361 27884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.67:41863 every 8 connection(s)
I20260928 06:40:35.361649 27756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:40:35 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577635, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:35.368510 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 27756
I20260928 06:40:35.368643 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance
I20260928 06:40:35.368852 19277 external_mini_cluster.cc:1428] Setting key 9e4ee2a59892e37a2035454bbdf28ab3
I20260928 06:40:35.370538 27466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:35.362367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:34775 (local address 127.18.211.126:34589)
0928 06:40:35.362438 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:35.362441 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:35.362582 (+ 141us) server_negotiation.cc:407] Connection header received
0928 06:40:35.363100 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:35.363101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:35.363115 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:35.363166 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:35.364076 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.364592 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.365557 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.365757 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.365819 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:35.367817 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:35.367819 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:35.367819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:35.367827 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:35.369303 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:35.369653 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:35.369654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:35.369654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:35.369704 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:35.369915 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:35.369915 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:35.369916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:35.370023 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:35.370077 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:35.370369 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 06:40:35.370451 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:40:35.370692 27887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:35.362289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.67:34775)
0928 06:40:35.362465 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:35.362505 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:35.362980 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:35.363238 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:35.363242 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:35.363790 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:35.363975 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:35.363980 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.364679 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.364680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:35.365471 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:35.365474 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.365833 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.365834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:35.365946 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:35.366405 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:35.366413 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:35.367725 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:35.369375 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:35.369381 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:35.369383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:35.369589 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:35.369759 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:35.369760 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:35.369760 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:35.369861 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:35.370076 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:35.370082 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:35.370211 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 06:40:35.370337 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 06:40:35.370485 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":94,"threads_started":1}
I20260928 06:40:35.371104 27885 heartbeater.cc:347] Connected to a master server at 127.18.211.126:34589
I20260928 06:40:35.371214 27885 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:35.371384 27885 heartbeater.cc:511] Master 127.18.211.126:34589 requested a full tablet report, sending...
I20260928 06:40:35.371747 27425 ts_manager.cc:194] Registered new tserver with Master: acbdd45f4d824decb21bbd60d0e5faca (127.18.211.67:41863)
I20260928 06:40:35.372099 27425 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.67@KRBTEST.COM'} at 127.18.211.67:34775
I20260928 06:40:35.380542 19277 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.18.211.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.211.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.211.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.65.keytab.
Entry for principal oryx/127.18.211.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.65.keytab.
WARNING: no policy specified for oryx/127.18.211.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.211.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.211.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.66.keytab.
Entry for principal oryx/127.18.211.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.66.keytab.
WARNING: no policy specified for oryx/127.18.211.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.211.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.211.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.67.keytab.
Entry for principal oryx/127.18.211.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.67.keytab.
I20260928 06:40:35.443625 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 27488
I20260928 06:40:35.447974 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 27622
I20260928 06:40:35.451629 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 27756
I20260928 06:40:35.455385 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 27394
I20260928 06:40:35.459548 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:34589
--webserver_interface=127.18.211.126
--webserver_port=41651
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:42391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:34589
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:35.537472 27894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:35.537683 27894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:35.537715 27894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:35.538854 27894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:40:35.538895 27894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:35.538913 27894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:35.538929 27894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:40:35.538944 27894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:40:35.540387 27894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:42391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:34589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:34589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=41651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:35.540630 27894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:35.540832 27894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:35.545414 27900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:35.545368 27899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:35.545383 27902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:35.545555 27894 server_base.cc:1034] running on GCE node
I20260928 06:40:35.545837 27894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:35.546080 27894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:35.547201 27894 hybrid_clock.cc:648] HybridClock initialized: now 1790577635547167 us; error 48 us; skew 500 ppm
Sep 28 06:40:35 dist-test-slave-b28r krb5kdc[27381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577635, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:35.549621 27894 init.cc:377] Logged in from keytab as kudu/127.18.211.126@KRBTEST.COM (short username kudu)
I20260928 06:40:35.550104 27894 webserver.cc:460] Webserver started at http://127.18.211.126:41651/ using document root <none> and password file <none>
I20260928 06:40:35.550282 27894 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:35.550350 27894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:35.551613 27894 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:35.552239 27909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:35.552383 27894 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:40:35.552453 27894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "2cd32525fadf45f8a212208cc8f98124"
format_stamp: "Formatted at 2026-09-28 06:40:34 on dist-test-slave-b28r"
server_key: "02629e0659cce1fda540c5d39958744c"
server_key_iv: "d96fb46970b130bcecb9845bb50d9596"
server_key_version: "encryptionkey@0"
I20260928 06:40:35.552754 27894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:35.572365 27894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:35.574132 27894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:35.574242 27894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:35.578068 27894 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:34589
I20260928 06:40:35.578116 27961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:34589 every 8 connection(s)
I20260928 06:40:35.578470 27894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:40:35.579932 27962 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:40:35.580536 27962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124: Bootstrap starting.
I20260928 06:40:35.583781 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 27894
I20260928 06:40:35.584062 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.65:41749
--local_ip_for_outbound_sockets=127.18.211.65
--tserver_master_addrs=127.18.211.126:34589
--webserver_port=34063
--webserver_interface=127.18.211.65
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:42391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.18.211.65
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:40:35.588670 27962 log.cc:824] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:35.590180 27962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:40:35.590420 27962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124: Bootstrap complete.
I20260928 06:40:35.592548 27962 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd32525fadf45f8a212208cc8f98124" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34589 } }
I20260928 06:40:35.592779 27962 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cd32525fadf45f8a212208cc8f98124, State: Initialized, Role: FOLLOWER
I20260928 06:40:35.592935 27962 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd32525fadf45f8a212208cc8f98124" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34589 } }
I20260928 06:40:35.593008 27962 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:40:35.593057 27962 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:40:35.593104 27962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124 [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:40:35.594020 27962 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd32525fadf45f8a212208cc8f98124" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34589 } }
I20260928 06:40:35.594111 27962 leader_election.cc:302] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cd32525fadf45f8a212208cc8f98124; no voters:
I20260928 06:40:35.594261 27962 leader_election.cc:288] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 06:40:35.594319 27966 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:40:35.594511 27962 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:40:35.594523 27966 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124 [term 2 LEADER]: Becoming Leader. State: Replica: 2cd32525fadf45f8a212208cc8f98124, State: Running, Role: LEADER
I20260928 06:40:35.594718 27966 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd32525fadf45f8a212208cc8f98124" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34589 } }
I20260928 06:40:35.595012 27967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2cd32525fadf45f8a212208cc8f98124" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd32525fadf45f8a212208cc8f98124" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34589 } } }
I20260928 06:40:35.595099 27967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124 [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:35.595320 27968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cd32525fadf45f8a212208cc8f98124. Latest consensus state: current_term: 2 leader_uuid: "2cd32525fadf45f8a212208cc8f98124" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cd32525fadf45f8a212208cc8f98124" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34589 } } }
I20260928 06:40:35.595420 27968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124 [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:35.595431 27974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:40:35.596004 27974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:40:35.596145 27974 catalog_manager.cc:1259] Loaded cluster ID: 003909d79cd74a4d83cc93be4aaa097b
I20260928 06:40:35.596203 27974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:40:35.597179 27974 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:40:35.597352 27974 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2cd32525fadf45f8a212208cc8f98124: Loaded TSK: 0
I20260928 06:40:35.597551 27974 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:40:35.679127 27964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:35.679322 27964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:35.679343 27964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:35.680507 27964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:35.680549 27964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:35.680575 27964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.65
I20260928 06:40:35.681975 27964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:42391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.65.keytab
--principal=oryx/127.18.211.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.65:41749
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.211.65
--webserver_port=34063
--tserver_master_addrs=127.18.211.126:34589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.65
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:35.682176 27964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:35.682395 27964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:35.687202 27988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:35.687317 27964 server_base.cc:1034] running on GCE node
W20260928 06:40:35.687211 27989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:35.687230 27991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:35.687667 27964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:35.687897 27964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:35.689035 27964 hybrid_clock.cc:648] HybridClock initialized: now 1790577635689025 us; error 33 us; skew 500 ppm
Sep 28 06:40:35 dist-test-slave-b28r krb5kdc[27381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577635, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:35.691529 27964 init.cc:377] Logged in from keytab as oryx/127.18.211.65@KRBTEST.COM (short username oryx)
I20260928 06:40:35.691963 27964 webserver.cc:460] Webserver started at http://127.18.211.65:34063/ using document root <none> and password file <none>
I20260928 06:40:35.692191 27964 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:35.692265 27964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:35.693538 27964 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:35.694095 27998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:35.694226 27964 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:35.694304 27964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-0/wal
uuid: "9e5b953aba534625adbab186dd9fab34"
format_stamp: "Formatted at 2026-09-28 06:40:35 on dist-test-slave-b28r"
server_key: "e1fa5ac77ddd5e10e00c29a61c6efa9a"
server_key_iv: "d517cb45efaadba7c1841a92ff465348"
server_key_version: "encryptionkey@0"
I20260928 06:40:35.694613 27964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:35.709503 27964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:35.711598 27964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:35.711742 27964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:35.711974 27964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:35.712298 27964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:35.712337 27964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:35.712365 27964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:35.712389 27964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:35.719100 27964 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.65:41749
I20260928 06:40:35.719142 28111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.65:41749 every 8 connection(s)
I20260928 06:40:35.719524 27964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:40:35 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:35 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:35.728646 28114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:35.720215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.65:43683)
0928 06:40:35.720392 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:35.720438 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:35.721032 (+ 594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:35.721643 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:35.721647 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:35.722205 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:35.722412 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:35.722417 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.724084 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.724085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:35.724864 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:35.724867 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.725645 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.725645 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:35.725743 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:35.726247 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:35.726255 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:35.728502 (+ 2247us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":92,"threads_started":1}
W20260928 06:40:35.728847 28115 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:35.720319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:43683 (local address 127.18.211.126:34589)
0928 06:40:35.721238 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:35.721247 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:35.721286 (+ 39us) server_negotiation.cc:407] Connection header received
0928 06:40:35.721410 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:35.721416 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:35.721468 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:35.721592 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:35.722530 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.723997 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.724946 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.725569 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.725671 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:35.728655 (+ 2984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.65:43683: BlockingRecv error: recv got EOF from 127.18.211.65:43683 (error 108)
Metrics: {"server-negotiator.queue_time_us":857,"thread_start_us":78,"threads_started":1}
W20260928 06:40:35.729085 28112 heartbeater.cc:644] Failed to heartbeat to 127.18.211.126:34589 (0 consecutive failures): Not authorized: Failed to ping master at 127.18.211.126:34589: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
I20260928 06:40:35.729537 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 27964
I20260928 06:40:35.729785 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.66:41497
--local_ip_for_outbound_sockets=127.18.211.66
--tserver_master_addrs=127.18.211.126:34589
--webserver_port=43801
--webserver_interface=127.18.211.66
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:42391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.18.211.66
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:40:35 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:35 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:35.735160 28114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:35.729355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.65:55411)
0928 06:40:35.730919 (+ 1564us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:35.730930 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:35.730983 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:35.731178 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:35.731179 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:35.731330 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:35.731386 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:35.731388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.732232 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.732233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:35.732731 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:35.732733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.733057 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.733057 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:35.733121 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:35.733514 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:35.733518 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:35.735075 (+ 1557us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1527}
W20260928 06:40:35.735253 28115 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:35.729387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:55411 (local address 127.18.211.126:34589)
0928 06:40:35.729423 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:35.729425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:35.730983 (+ 1558us) server_negotiation.cc:407] Connection header received
0928 06:40:35.731061 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:35.731062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:35.731076 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:35.731135 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:35.731467 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.732155 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.732799 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.733000 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.733053 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:35.735194 (+ 2141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.65:55411: BlockingRecv error: recv got EOF from 127.18.211.65:55411 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 28 06:40:35 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:35 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:35.739300 28114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:35.735605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.65:35231)
0928 06:40:35.735691 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:35.735701 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:35.735743 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:35.735863 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:35.735864 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:35.735977 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:35.736001 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:35.736002 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.736565 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.736566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:35.737048 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:35.737049 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.737347 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.737348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:35.737395 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:35.737747 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:35.737752 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:35.739237 (+ 1485us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":72}
W20260928 06:40:35.739364 28115 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:35.735657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:35231 (local address 127.18.211.126:34589)
0928 06:40:35.735693 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:35.735694 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:35.735731 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:40:35.735794 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:35.735795 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:35.735804 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:35.735856 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:35.736060 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.736500 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.737112 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.737290 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.737343 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:35.739314 (+ 1971us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.65:35231: BlockingRecv error: recv got EOF from 127.18.211.65:35231 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260928 06:40:35.739656 28112 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:40:35.809789 28116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:35.809978 28116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:35.809998 28116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:35.811124 28116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:35.811160 28116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:35.811184 28116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.66
I20260928 06:40:35.812528 28116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:42391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.66.keytab
--principal=oryx/127.18.211.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.66:41497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.211.66
--webserver_port=43801
--tserver_master_addrs=127.18.211.126:34589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.66
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:35.812695 28116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:35.812866 28116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:35.817411 28124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:35.817446 28122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:35.817447 28116 server_base.cc:1034] running on GCE node
W20260928 06:40:35.817416 28121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:35.817804 28116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:35.818035 28116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:35.819141 28116 hybrid_clock.cc:648] HybridClock initialized: now 1790577635819124 us; error 27 us; skew 500 ppm
Sep 28 06:40:35 dist-test-slave-b28r krb5kdc[27381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577635, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:35.821445 28116 init.cc:377] Logged in from keytab as oryx/127.18.211.66@KRBTEST.COM (short username oryx)
I20260928 06:40:35.821869 28116 webserver.cc:460] Webserver started at http://127.18.211.66:43801/ using document root <none> and password file <none>
I20260928 06:40:35.822058 28116 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:35.822116 28116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:35.823487 28116 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:35.824018 28131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:35.824160 28116 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:35.824265 28116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-1/wal
uuid: "e4257cb77c944960bbb6018e968f65c7"
format_stamp: "Formatted at 2026-09-28 06:40:35 on dist-test-slave-b28r"
server_key: "37e9f707b95a0b1a7f8dc120bc07ff16"
server_key_iv: "3bee3e9b6d547cd8cec51167112d13b5"
server_key_version: "encryptionkey@0"
I20260928 06:40:35.824570 28116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:35.837893 28116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:35.839895 28116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:35.840029 28116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:35.840257 28116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:35.840549 28116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:35.840581 28116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:35.840610 28116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:35.840627 28116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:35.846124 28116 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.66:41497
I20260928 06:40:35.846180 28244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.66:41497 every 8 connection(s)
I20260928 06:40:35.846489 28116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:40:35 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:35 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 06:40:35.853490 28115 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:35.847158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:53581 (local address 127.18.211.126:34589)
0928 06:40:35.847206 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:35.847207 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:35.847445 (+ 238us) server_negotiation.cc:407] Connection header received
0928 06:40:35.847983 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:35.847985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:35.848002 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:35.848052 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:35.848964 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.849455 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.850432 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.850639 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.850695 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:35.853434 (+ 2739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.66:53581: BlockingRecv error: recv got EOF from 127.18.211.66:53581 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:40:35.853526 28247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:35.847145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.66:53581)
0928 06:40:35.847317 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:35.847355 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:35.847866 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:35.848124 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:35.848128 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:35.848665 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:35.848862 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:35.848867 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.849540 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.849541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:35.850341 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:35.850343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.850708 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.850709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:35.850835 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:35.851360 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:35.851369 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:35.853335 (+ 1966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
W20260928 06:40:35.853816 28245 heartbeater.cc:644] Failed to heartbeat to 127.18.211.126:34589 (0 consecutive failures): Not authorized: Failed to ping master at 127.18.211.126:34589: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
I20260928 06:40:35.854750 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 28116
I20260928 06:40:35.855038 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.67:41863
--local_ip_for_outbound_sockets=127.18.211.67
--tserver_master_addrs=127.18.211.126:34589
--webserver_port=45635
--webserver_interface=127.18.211.67
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:42391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.18.211.67
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:40:35 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:35 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:35.858494 28247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:35.854015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.66:60251)
0928 06:40:35.854056 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:35.854078 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:35.854116 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:35.854222 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:35.854222 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:35.854350 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:35.854394 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:35.854396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.855264 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.855265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:35.855758 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:35.855760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.856147 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.856148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:35.856211 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:35.856587 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:35.856592 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:35.858429 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 06:40:35.858585 28115 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:35.854066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:60251 (local address 127.18.211.126:34589)
0928 06:40:35.854122 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:35.854124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:35.854133 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:40:35.854158 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:35.854159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:35.854168 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:35.854215 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:35.854462 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.855174 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.855829 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.856082 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.856149 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:35.858525 (+ 2376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.66:60251: BlockingRecv error: recv got EOF from 127.18.211.66:60251 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 28 06:40:35 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:35 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:35.862629 28247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:35.858854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.66:57973)
0928 06:40:35.858919 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:35.858928 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:35.858964 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:35.859066 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:35.859067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:35.859201 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:35.859226 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:35.859227 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.859794 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.859795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:35.860287 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:35.860289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.860616 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.860618 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:35.860667 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:35.861009 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:35.861014 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:35.862586 (+ 1572us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 06:40:35.862752 28115 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:35.858919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:57973 (local address 127.18.211.126:34589)
0928 06:40:35.858971 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:35.858973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:35.858983 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:40:35.859003 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:35.859004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:35.859013 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:35.859059 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:35.859287 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.859730 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.860355 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.860557 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.860612 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:35.862686 (+ 2074us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.66:57973: BlockingRecv error: recv got EOF from 127.18.211.66:57973 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260928 06:40:35.862845 28245 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:40:35.936832 28248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:35.937017 28248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:35.937039 28248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:35.938145 28248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:35.938185 28248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:35.938210 28248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.67
I20260928 06:40:35.939592 28248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:42391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/krb5kdc/oryx_127.18.211.67.keytab
--principal=oryx/127.18.211.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.67:41863
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.211.67
--webserver_port=45635
--tserver_master_addrs=127.18.211.126:34589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.67
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:35.939802 28248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:35.939996 28248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:35.944525 28256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:35.944533 28254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:35.944597 28248 server_base.cc:1034] running on GCE node
W20260928 06:40:35.944536 28253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:35.944916 28248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:35.945130 28248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:35.946269 28248 hybrid_clock.cc:648] HybridClock initialized: now 1790577635946238 us; error 43 us; skew 500 ppm
Sep 28 06:40:35 dist-test-slave-b28r krb5kdc[27381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577635, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.211.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:35.948652 28248 init.cc:377] Logged in from keytab as oryx/127.18.211.67@KRBTEST.COM (short username oryx)
I20260928 06:40:35.949061 28248 webserver.cc:460] Webserver started at http://127.18.211.67:45635/ using document root <none> and password file <none>
I20260928 06:40:35.949258 28248 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:35.949323 28248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:35.950659 28248 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:35.951277 28263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:35.951464 28248 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:40:35.951560 28248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-2/wal
uuid: "acbdd45f4d824decb21bbd60d0e5faca"
format_stamp: "Formatted at 2026-09-28 06:40:35 on dist-test-slave-b28r"
server_key: "b464c88fb2b8c9500a1f6f6197d8a099"
server_key_iv: "22e3c3fb288d7e29ff5a125eebbf7d80"
server_key_version: "encryptionkey@0"
I20260928 06:40:35.951892 28248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:35.964110 28248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:35.966058 28248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:35.966183 28248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:35.966374 28248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:35.966634 28248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:35.966665 28248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:35.966691 28248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:35.966724 28248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:35.972234 28248 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.67:41863
I20260928 06:40:35.972270 28376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.67:41863 every 8 connection(s)
I20260928 06:40:35.972573 28248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:40:35 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:35 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 06:40:35.979386 28115 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:35.973254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:57999 (local address 127.18.211.126:34589)
0928 06:40:35.973305 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:35.973307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:35.973525 (+ 218us) server_negotiation.cc:407] Connection header received
0928 06:40:35.974002 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:35.974003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:35.974014 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:35.974057 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:35.974953 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.975524 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.976479 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.976657 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.976702 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:35.979339 (+ 2637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.67:57999: BlockingRecv error: recv got EOF from 127.18.211.67:57999 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:40:35.979391 28379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:35.973211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.67:57999)
0928 06:40:35.973362 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:35.973405 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:35.973902 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:35.974111 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:35.974114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:35.974645 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:35.974843 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:35.974848 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.975624 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.975625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:35.976408 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:35.976410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.976738 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.976739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:35.976856 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:35.977305 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:35.977314 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:35.979252 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
W20260928 06:40:35.979635 28377 heartbeater.cc:644] Failed to heartbeat to 127.18.211.126:34589 (0 consecutive failures): Not authorized: Failed to ping master at 127.18.211.126:34589: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
I20260928 06:40:35.980139 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 28248
Sep 28 06:40:35 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:35 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:35.984234 28379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:35.979833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.67:47959)
0928 06:40:35.979872 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:35.979882 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:35.979918 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:35.980105 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:35.980105 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:35.980231 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:35.980284 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:35.980286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.981107 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.981108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:35.981661 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:35.981663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.982050 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.982050 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:35.982113 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:35.982457 (+ 344us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:35.982461 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:35.984153 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 06:40:35.984308 28115 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:35.979981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:47959 (local address 127.18.211.126:34589)
0928 06:40:35.980013 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:35.980014 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:35.980021 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:40:35.980040 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:35.980041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:35.980048 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:35.980090 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:35.980351 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.981023 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.981797 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.981984 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.982042 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:35.984251 (+ 2209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.67:47959: BlockingRecv error: recv got EOF from 127.18.211.67:47959 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
Sep 28 06:40:35 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:35.987571 28380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:35.980595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59518 (local address 127.18.211.126:34589)
0928 06:40:35.980695 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:35.980697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:35.980793 (+ 96us) server_negotiation.cc:407] Connection header received
0928 06:40:35.980853 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:35.980853 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:35.980861 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:35.980920 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:35.981529 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.982113 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.982915 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.983144 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.983204 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:35.984589 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:35.984601 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:35.984601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:35.984613 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:35.986097 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:35.986462 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:35.986466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:35.986466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:35.986579 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:35.986792 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:35.986793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:35.986794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:35.987122 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:35.987258 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:35.987371 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 06:40:35.987475 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":55,"threads_started":1}
Sep 28 06:40:35 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:35.988556 28379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:35.984444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.67:47097)
0928 06:40:35.984547 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:35.984555 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:35.984615 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:35.984754 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:35.984755 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:35.984855 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:35.984880 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:35.984881 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.985461 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.985462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:35.985983 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:35.985985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.986336 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.986337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:35.986381 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:35.986697 (+ 316us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:35.986701 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:35.988497 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":68,"threads_started":1}
W20260928 06:40:35.988627 28115 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:35.984498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:47097 (local address 127.18.211.126:34589)
0928 06:40:35.984533 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:35.984535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:35.984592 (+ 57us) server_negotiation.cc:407] Connection header received
0928 06:40:35.984668 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:35.984668 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:35.984676 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:35.984730 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:35.984958 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.985394 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.986051 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:35.986272 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:35.986339 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:35.988580 (+ 2241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.67:47097: BlockingRecv error: recv got EOF from 127.18.211.67:47097 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 06:40:35.988687 28377 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 06:40:36 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:36 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:36.747308 28383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:36.740030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.65:53471)
0928 06:40:36.740148 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:36.740161 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:36.740235 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:36.740404 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:36.740404 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:36.740555 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:36.740603 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:36.740605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:36.741252 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:36.741253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:36.741820 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:36.741822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:36.742171 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:36.742172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:36.742210 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:36.745593 (+ 3383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:36.745598 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:36.747243 (+ 1645us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
W20260928 06:40:36.747421 28384 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:36.740136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:53471 (local address 127.18.211.126:34589)
0928 06:40:36.740296 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:36.740298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:36.740306 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:40:36.740326 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:36.740326 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:36.740336 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:36.740390 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:36.740655 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:36.741135 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:36.741886 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:36.742116 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:36.742167 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:36.747355 (+ 5188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.65:53471: BlockingRecv error: recv got EOF from 127.18.211.65:53471 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":72,"threads_started":1}
Sep 28 06:40:36 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:36 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:36.867764 28385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:36.863241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.66:59881)
0928 06:40:36.863374 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:36.863386 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:36.863455 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:36.863578 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:36.863579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:36.863746 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:36.863815 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:36.863817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:36.864464 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:36.864465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:36.865016 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:36.865018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:36.865352 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:36.865353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:36.865409 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:36.865873 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:36.865878 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:36.867686 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
W20260928 06:40:36.867863 28384 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:36.863303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:59881 (local address 127.18.211.126:34589)
0928 06:40:36.863381 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:36.863383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:36.863435 (+ 52us) server_negotiation.cc:407] Connection header received
0928 06:40:36.863505 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:36.863506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:36.863517 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:36.863568 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:36.863890 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:36.864394 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:36.865092 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:36.865292 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:36.865347 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:36.867802 (+ 2455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.66:59881: BlockingRecv error: recv got EOF from 127.18.211.66:59881 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 06:40:36 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:36 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:36.993721 28386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:36.989062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.67:39791)
0928 06:40:36.989197 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:36.989209 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:36.989307 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:36.989441 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:36.989441 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:36.989602 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:36.989648 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:36.989650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:36.990486 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:36.990487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:36.991040 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:36.991042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:36.991421 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:36.991422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:36.991476 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:36.991873 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:36.991877 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:36.993644 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":57,"threads_started":1}
W20260928 06:40:36.993820 28384 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:36.989136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:39791 (local address 127.18.211.126:34589)
0928 06:40:36.989198 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:36.989200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:36.989306 (+ 106us) server_negotiation.cc:407] Connection header received
0928 06:40:36.989362 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:36.989363 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:36.989377 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:36.989427 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:36.989734 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:36.990412 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:36.991143 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:36.991353 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:36.991408 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:36.993760 (+ 2352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.67:39791: BlockingRecv error: recv got EOF from 127.18.211.67:39791 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 28 06:40:37 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:37 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:37.751946 28387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:37.747714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.65:60955)
0928 06:40:37.747830 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:37.747840 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:37.747899 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:37.748025 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:37.748025 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:37.748174 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:37.748222 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:37.748224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:37.748877 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:37.748877 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:37.749406 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:37.749407 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:37.749747 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:37.749748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:37.749799 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:37.750221 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:37.750226 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:37.751889 (+ 1663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":63,"threads_started":1}
W20260928 06:40:37.752059 28388 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:37.747779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:60955 (local address 127.18.211.126:34589)
0928 06:40:37.747915 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:37.747917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:37.747925 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:40:37.747945 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:37.747946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:37.747955 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:37.748016 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:37.748315 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:37.748805 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:37.749485 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:37.749683 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:37.749738 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:37.751981 (+ 2243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.65:60955: BlockingRecv error: recv got EOF from 127.18.211.65:60955 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":62,"threads_started":1}
Sep 28 06:40:37 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:37 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:37.872351 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:37.868226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.66:39621)
0928 06:40:37.868360 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:37.868370 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:37.868411 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:37.868524 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:37.868524 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:37.868677 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:37.868727 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:37.868729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:37.869326 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:37.869327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:37.869868 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:37.869870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:37.870147 (+ 277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:37.870148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:37.870184 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:37.870623 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:37.870628 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:37.872291 (+ 1663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":66,"threads_started":1}
W20260928 06:40:37.872510 28388 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:37.868317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:39621 (local address 127.18.211.126:34589)
0928 06:40:37.868390 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:37.868392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:37.868421 (+ 29us) server_negotiation.cc:407] Connection header received
0928 06:40:37.868464 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:37.868465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:37.868476 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:37.868518 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:37.868777 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:37.869262 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:37.869924 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:37.870103 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:37.870143 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:37.872438 (+ 2295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.66:39621: BlockingRecv error: recv got EOF from 127.18.211.66:39621 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 06:40:37 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:37 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:37.998854 28390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:37.994188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.67:49131)
0928 06:40:37.994330 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:37.994343 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:37.994409 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:37.994567 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:37.994568 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:37.994840 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:37.994905 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:37.994907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:37.995624 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:37.995625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:37.996243 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:37.996245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:37.996586 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:37.996587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:37.996654 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:37.997089 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:37.997093 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:37.998796 (+ 1703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":68,"threads_started":1}
W20260928 06:40:37.998942 28388 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:37.994340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:49131 (local address 127.18.211.126:34589)
0928 06:40:37.994400 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:37.994402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:37.994412 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:40:37.994471 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:37.994471 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:37.994486 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:37.994546 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:37.994971 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:37.995534 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:37.996331 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:37.996514 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:37.996564 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:37.998885 (+ 2321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.67:49131: BlockingRecv error: recv got EOF from 127.18.211.67:49131 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 28 06:40:38 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:38 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:38.756852 28391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:38.752386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.65:55371)
0928 06:40:38.752498 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:38.752508 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:38.752555 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:38.752725 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:38.752725 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:38.752877 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:38.752925 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:38.752927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:38.753554 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:38.753555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:38.754115 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:38.754117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:38.754450 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:38.754451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:38.754505 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:38.754896 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:38.754900 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:38.756728 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":57,"threads_started":1}
W20260928 06:40:38.756893 28392 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:38.752466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:55371 (local address 127.18.211.126:34589)
0928 06:40:38.752581 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:38.752583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:38.752590 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:40:38.752610 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:38.752611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:38.752621 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:38.752696 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:38.752993 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:38.753483 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:38.754184 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:38.754390 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:38.754445 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:38.756831 (+ 2386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.65:55371: BlockingRecv error: recv got EOF from 127.18.211.65:55371 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":58,"threads_started":1}
Sep 28 06:40:38 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:38 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:38.877221 28393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:38.872781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.66:40217)
0928 06:40:38.872926 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:38.872938 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:38.873003 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:38.873145 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:38.873146 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:38.873330 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:38.873376 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:38.873379 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:38.874054 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:38.874055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:38.874573 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:38.874575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:38.874901 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:38.874901 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:38.874958 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:38.875418 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:38.875422 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:38.877160 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":69,"threads_started":1}
W20260928 06:40:38.877357 28392 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:38.872879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:40217 (local address 127.18.211.126:34589)
0928 06:40:38.872930 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:38.872933 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:38.872994 (+ 61us) server_negotiation.cc:407] Connection header received
0928 06:40:38.873066 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:38.873067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:38.873080 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:38.873124 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:38.873448 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:38.873977 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:38.874650 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:38.874838 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:38.874893 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:38.877288 (+ 2395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.66:40217: BlockingRecv error: recv got EOF from 127.18.211.66:40217 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 28 06:40:38 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:39 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:39.003774 28394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:38.999365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.67:46599)
0928 06:40:38.999511 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:38.999523 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:38.999588 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:38.999710 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:38.999711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:38.999872 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:38.999924 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:38.999926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:39.000549 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:39.000550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:39.001080 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:39.001081 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:39.001417 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:39.001418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:39.001458 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:39.001865 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:39.001869 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:39.003713 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":71,"threads_started":1}
W20260928 06:40:39.003866 28392 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:38.999426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:46599 (local address 127.18.211.126:34589)
0928 06:40:38.999477 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:38.999479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:38.999579 (+ 100us) server_negotiation.cc:407] Connection header received
0928 06:40:38.999641 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:38.999642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:38.999653 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:38.999700 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:38.999980 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:39.000486 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:39.001153 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:39.001360 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:39.001410 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:39.003807 (+ 2397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.67:46599: BlockingRecv error: recv got EOF from 127.18.211.67:46599 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 28 06:40:39 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:39 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:39.762059 28395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:39.757341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.65:49553)
0928 06:40:39.757471 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:39.757480 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:39.757527 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:39.757695 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:39.757695 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:39.757852 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:39.757901 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:39.757902 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:39.758662 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:39.758663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:39.759359 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:39.759363 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:39.759749 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:39.759750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:39.759799 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:39.760275 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:39.760280 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:39.761972 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":59,"threads_started":1}
W20260928 06:40:39.762144 28396 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:39.757446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:49553 (local address 127.18.211.126:34589)
0928 06:40:39.757565 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:39.757566 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:39.757575 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:40:39.757597 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:39.757598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:39.757609 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:39.757681 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:39.757989 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:39.758549 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:39.759450 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:39.759653 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:39.759713 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:39.762078 (+ 2365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.65:49553: BlockingRecv error: recv got EOF from 127.18.211.65:49553 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
Sep 28 06:40:39 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:39 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:39.882277 28397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:39.877755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.66:52845)
0928 06:40:39.877929 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:39.877940 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:39.878021 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:39.878138 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:39.878139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:39.878307 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:39.878357 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:39.878359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:39.879042 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:39.879043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:39.879609 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:39.879611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:39.879974 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:39.879975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:39.880025 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:39.880458 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:39.880463 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:39.882208 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":84,"threads_started":1}
W20260928 06:40:39.882373 28396 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:39.877879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:52845 (local address 127.18.211.126:34589)
0928 06:40:39.877929 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:39.877932 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:39.878015 (+ 83us) server_negotiation.cc:407] Connection header received
0928 06:40:39.878066 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:39.878067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:39.878077 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:39.878126 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:39.878469 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:39.878972 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:39.879689 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:39.879884 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:39.879949 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:39.882308 (+ 2359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.66:52845: BlockingRecv error: recv got EOF from 127.18.211.66:52845 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 28 06:40:40 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:40 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:40.008450 28398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:40.004217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.67:57615)
0928 06:40:40.004349 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:40.004361 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:40.004426 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:40.004546 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:40.004547 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:40.004701 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:40.004751 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:40.004753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:40.005383 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:40.005384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:40.005919 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:40.005921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:40.006216 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:40.006217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:40.006251 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:40.006663 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:40.006667 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:40.008390 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
W20260928 06:40:40.008556 28396 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:40.004310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:57615 (local address 127.18.211.126:34589)
0928 06:40:40.004382 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:40.004384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:40.004410 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:40:40.004477 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:40.004477 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:40.004489 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:40.004535 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:40.004821 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:40.005303 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:40.005977 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:40.006167 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:40.006214 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:40.008484 (+ 2270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.67:57615: BlockingRecv error: recv got EOF from 127.18.211.67:57615 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 06:40:40 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:40 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:40.766932 28399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:40.762529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.65:33629)
0928 06:40:40.762692 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:40.762703 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:40.762762 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:40.762908 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:40.762909 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:40.763065 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:40.763141 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:40.763144 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:40.763841 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:40.763842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:40.764381 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:40.764383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:40.764680 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:40.764681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:40.764718 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:40.765126 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:40.765130 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:40.766872 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":115,"threads_started":1}
W20260928 06:40:40.767035 28400 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:40.762616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:33629 (local address 127.18.211.126:34589)
0928 06:40:40.762732 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:40.762734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:40.762748 (+ 14us) server_negotiation.cc:407] Connection header received
0928 06:40:40.762825 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:40.762826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:40.762837 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:40.762891 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:40.763246 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:40.763761 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:40.764444 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:40.764631 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:40.764676 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:40.766967 (+ 2291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.65:33629: BlockingRecv error: recv got EOF from 127.18.211.65:33629 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":61,"threads_started":1}
Sep 28 06:40:40 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:40 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:40.887411 28401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:40.882814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.66:49693)
0928 06:40:40.882958 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:40.882971 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:40.883033 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:40.883229 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:40.883230 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:40.883392 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:40.883437 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:40.883438 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:40.884056 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:40.884057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:40.884579 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:40.884580 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:40.884931 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:40.884932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:40.884976 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:40.885456 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:40.885461 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:40.887352 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":60,"threads_started":1}
W20260928 06:40:40.887480 28400 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:40.882997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:49693 (local address 127.18.211.126:34589)
0928 06:40:40.883059 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:40.883061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:40.883070 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:40:40.883116 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:40.883117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:40.883127 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:40.883189 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:40.883511 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:40.883996 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:40.884669 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:40.884868 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:40.884922 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:40.887428 (+ 2506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.66:49693: BlockingRecv error: recv got EOF from 127.18.211.66:49693 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 28 06:40:41 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:41 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:41.013290 28402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:41.008905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.67:47413)
0928 06:40:41.009054 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:41.009064 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:41.009110 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:41.009283 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:41.009284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:41.009440 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:41.009490 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:41.009492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:41.010115 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:41.010116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:41.010654 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:41.010656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:41.011013 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:41.011013 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:41.011048 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:41.011524 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:41.011528 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:41.013231 (+ 1703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":76,"threads_started":1}
W20260928 06:40:41.013429 28400 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:41.009036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:47413 (local address 127.18.211.126:34589)
0928 06:40:41.009095 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:41.009097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:41.009106 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:40:41.009153 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:41.009154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:41.009167 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:41.009274 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:41.009552 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:41.010054 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:41.010753 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:41.010950 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:41.011011 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:41.013360 (+ 2349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.67:47413: BlockingRecv error: recv got EOF from 127.18.211.67:47413 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 28 06:40:41 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:41 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:41.776060 28403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:41.767434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.65:33305)
0928 06:40:41.767598 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:41.767609 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:41.767669 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:41.767816 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:41.767816 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:41.767989 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:41.768038 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:41.768040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:41.769171 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:41.769172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:41.769725 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:41.769727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:41.770058 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:41.770058 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:41.770094 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:41.774158 (+ 4064us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:41.774164 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:41.776002 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":107,"threads_started":1}
W20260928 06:40:41.776163 28404 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:41.767519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:33305 (local address 127.18.211.126:34589)
0928 06:40:41.767625 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:41.767627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:41.767655 (+ 28us) server_negotiation.cc:407] Connection header received
0928 06:40:41.767736 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:41.767737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:41.767749 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:41.767813 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:41.768582 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:41.769085 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:41.769805 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:41.770002 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:41.770055 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:41.776096 (+ 6041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.65:33305: BlockingRecv error: recv got EOF from 127.18.211.65:33305 (error 108)
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":55,"threads_started":1}
Sep 28 06:40:41 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:41 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:41.898602 28405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:41.887860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.66:33127)
0928 06:40:41.887997 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:41.888007 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:41.888064 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:41.888184 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:41.888185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:41.888358 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:41.888405 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:41.888407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:41.889056 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:41.889057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:41.889827 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:41.889830 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:41.895423 (+ 5593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:41.895424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:41.895501 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:41.895994 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:41.896036 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:41.898503 (+ 2467us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":80,"threads_started":1}
W20260928 06:40:41.898749 28404 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:41.887943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:33127 (local address 127.18.211.126:34589)
0928 06:40:41.888010 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:41.888012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:41.888051 (+ 39us) server_negotiation.cc:407] Connection header received
0928 06:40:41.888120 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:41.888121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:41.888131 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:41.888175 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:41.888463 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:41.888986 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:41.894919 (+ 5933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:41.895228 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:41.895420 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:41.898618 (+ 3198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.66:33127: BlockingRecv error: recv got EOF from 127.18.211.66:33127 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 06:40:42 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:42 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:42.018551 28406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:42.013778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.67:48087)
0928 06:40:42.013957 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:42.013969 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:42.014044 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:42.014183 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:42.014184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:42.014355 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:42.014404 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:42.014406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:42.015219 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:42.015220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:42.015755 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:42.015757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:42.016113 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:42.016114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:42.016156 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:42.016628 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:42.016632 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:42.018492 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":94,"threads_started":1}
W20260928 06:40:42.018651 28404 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:42.013864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:48087 (local address 127.18.211.126:34589)
0928 06:40:42.013953 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:42.013955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:42.014033 (+ 78us) server_negotiation.cc:407] Connection header received
0928 06:40:42.014106 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:42.014107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:42.014122 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:42.014170 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:42.014488 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:42.015142 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:42.015852 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:42.016044 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:42.016108 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:42.018589 (+ 2481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.67:48087: BlockingRecv error: recv got EOF from 127.18.211.67:48087 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 06:40:42 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:42 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:42.780931 28407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:42.776545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.65:56787)
0928 06:40:42.776686 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:42.776697 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:42.776745 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:42.776915 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:42.776916 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:42.777075 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:42.777125 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:42.777127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:42.777799 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:42.777800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:42.778330 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:42.778331 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:42.778682 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:42.778683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:42.778717 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:42.779188 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:42.779193 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:42.780871 (+ 1678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":69,"threads_started":1}
W20260928 06:40:42.781068 28408 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:42.776652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:56787 (local address 127.18.211.126:34589)
0928 06:40:42.776792 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:42.776794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:42.776801 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:40:42.776822 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:42.776823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:42.776832 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:42.776894 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:42.777233 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:42.777733 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:42.778416 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:42.778621 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:42.778681 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:42.780985 (+ 2304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.65:56787: BlockingRecv error: recv got EOF from 127.18.211.65:56787 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
Sep 28 06:40:42 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:42 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:42.904187 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:42.899033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.66:42517)
0928 06:40:42.899256 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:42.899268 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:42.899336 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:42.899474 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:42.899475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:42.899647 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:42.899696 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:42.899698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:42.900346 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:42.900347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:42.900878 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:42.900879 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:42.901218 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:42.901219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:42.901275 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:42.901736 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:42.901741 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:42.904096 (+ 2355us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":96,"threads_started":1}
W20260928 06:40:42.904247 28408 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:42.899133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:42517 (local address 127.18.211.126:34589)
0928 06:40:42.899196 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:42.899199 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:42.899330 (+ 131us) server_negotiation.cc:407] Connection header received
0928 06:40:42.899401 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:42.899402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:42.899415 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:42.899463 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:42.899788 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:42.900275 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:42.900953 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:42.901145 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:42.901199 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:42.904193 (+ 2994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.66:42517: BlockingRecv error: recv got EOF from 127.18.211.66:42517 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 06:40:43 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:43 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:43.024025 28410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:43.019149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.67:33333)
0928 06:40:43.019274 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:43.019285 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:43.019356 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:43.019501 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:43.019501 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:43.019663 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:43.019716 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:43.019717 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:43.020410 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:43.020411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:43.020953 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:43.020954 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:43.021416 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:43.021418 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:43.021476 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:43.021986 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:43.021990 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:43.023957 (+ 1967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
W20260928 06:40:43.024111 28408 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:43.019262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:33333 (local address 127.18.211.126:34589)
0928 06:40:43.019311 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:43.019312 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:43.019339 (+ 27us) server_negotiation.cc:407] Connection header received
0928 06:40:43.019423 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:43.019423 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:43.019435 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:43.019474 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:43.019775 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:43.020268 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:43.021068 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:43.021314 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:43.021381 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:43.024048 (+ 2667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.67:33333: BlockingRecv error: recv got EOF from 127.18.211.67:33333 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 28 06:40:43 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:43 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:43.786033 28411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:43.781446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.65:41167)
0928 06:40:43.781597 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:43.781609 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:43.781667 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:43.781823 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:43.781824 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:43.782020 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:43.782077 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:43.782079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:43.782762 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:43.782763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:43.783367 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:43.783370 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:43.783702 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:43.783703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:43.783765 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:43.784222 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:43.784226 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:43.785965 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":63,"threads_started":1}
W20260928 06:40:43.786127 28412 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:43.781549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:41167 (local address 127.18.211.126:34589)
0928 06:40:43.781669 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:43.781671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:43.781699 (+ 28us) server_negotiation.cc:407] Connection header received
0928 06:40:43.781720 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:43.781720 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:43.781732 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:43.781804 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:43.782148 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:43.782680 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:43.783437 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:43.783631 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:43.783683 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:43.786056 (+ 2373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.65:41167: BlockingRecv error: recv got EOF from 127.18.211.65:41167 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":61,"threads_started":1}
Sep 28 06:40:43 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:43 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:43.909873 28413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:43.905508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.66:59471)
0928 06:40:43.905625 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:43.905635 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:43.905688 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:43.905805 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:43.905806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:43.905997 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:43.906047 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:43.906049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:43.906816 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:43.906817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:43.907380 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:43.907383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:43.907687 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:43.907688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:43.907732 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:43.908136 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:43.908140 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:43.909813 (+ 1673us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":62,"threads_started":1}
W20260928 06:40:43.909956 28412 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:43.905577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:59471 (local address 127.18.211.126:34589)
0928 06:40:43.905640 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:43.905642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:43.905674 (+ 32us) server_negotiation.cc:407] Connection header received
0928 06:40:43.905728 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:43.905729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:43.905740 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:43.905792 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:43.906111 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:43.906745 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:43.907446 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:43.907634 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:43.907678 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:43.909898 (+ 2220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.66:59471: BlockingRecv error: recv got EOF from 127.18.211.66:59471 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 06:40:44 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:44 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:44.029106 28414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:44.024521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.67:49611)
0928 06:40:44.024685 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:44.024698 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:44.024777 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:44.024902 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:44.024903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:44.025072 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:44.025118 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:44.025119 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:44.025808 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:44.025809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:44.026327 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:44.026328 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:44.026646 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:44.026647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:44.026702 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:44.027152 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:44.027157 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:44.029033 (+ 1876us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":77,"threads_started":1}
W20260928 06:40:44.029327 28412 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:44.024610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:49611 (local address 127.18.211.126:34589)
0928 06:40:44.024689 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:44.024692 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:44.024758 (+ 66us) server_negotiation.cc:407] Connection header received
0928 06:40:44.024839 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:44.024840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:44.024854 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:44.024896 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:44.025184 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:44.025715 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:44.026398 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:44.026587 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:44.026642 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:44.029183 (+ 2541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.67:49611: BlockingRecv error: recv got EOF from 127.18.211.67:49611 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 06:40:44 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:44 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:44.790921 28415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:44.786490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.65:50527)
0928 06:40:44.786609 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:44.786621 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:44.786689 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:44.786838 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:44.786839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:44.787016 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:44.787063 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:44.787065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:44.787745 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:44.787746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:44.788278 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:44.788280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:44.788611 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:44.788611 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:44.788667 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:44.789110 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:44.789115 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:44.790864 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":67,"threads_started":1}
W20260928 06:40:44.791050 28416 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:44.786563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:50527 (local address 127.18.211.126:34589)
0928 06:40:44.786689 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:44.786691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:44.786699 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:40:44.786746 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:44.786747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:44.786759 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:44.786828 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:44.787160 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:44.787669 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:44.788352 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:44.788552 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:44.788604 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:44.790975 (+ 2371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.65:50527: BlockingRecv error: recv got EOF from 127.18.211.65:50527 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
Sep 28 06:40:44 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:44 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:44.914928 28417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:44.910336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.66:41423)
0928 06:40:44.910473 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:44.910484 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:44.910542 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:44.910745 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:44.910746 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:44.910933 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:44.910982 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:44.910983 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:44.911669 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:44.911670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:44.912229 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:44.912231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:44.912611 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:44.912612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:44.912655 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:44.913129 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:44.913133 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:44.914859 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":59,"threads_started":1}
W20260928 06:40:44.915036 28416 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:44.910410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:41423 (local address 127.18.211.126:34589)
0928 06:40:44.910462 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:44.910463 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:44.910584 (+ 121us) server_negotiation.cc:407] Connection header received
0928 06:40:44.910615 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:44.910616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:44.910628 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:44.910672 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:44.911043 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:44.911586 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:44.912344 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:44.912547 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:44.912602 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:44.914971 (+ 2369us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.66:41423: BlockingRecv error: recv got EOF from 127.18.211.66:41423 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 28 06:40:45 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:45 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:45.034443 28418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:45.029550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.67:46873)
0928 06:40:45.029686 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:45.029699 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:45.029743 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:45.029918 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:45.029919 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:45.030074 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:45.030123 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:45.030125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:45.030820 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:45.030821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:45.031396 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:45.031398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:45.031785 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:45.031786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:45.031825 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:45.032361 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:45.032366 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:45.034348 (+ 1982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
W20260928 06:40:45.034515 28416 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:45.029630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:46873 (local address 127.18.211.126:34589)
0928 06:40:45.029717 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:45.029719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:45.029756 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:40:45.029836 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:45.029837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:45.029852 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:45.029900 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:45.030242 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:45.030742 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:45.031487 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:45.031679 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:45.031764 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:45.034458 (+ 2694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.67:46873: BlockingRecv error: recv got EOF from 127.18.211.67:46873 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 06:40:45 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:45 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:45.796020 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:45.791404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.65:42829)
0928 06:40:45.791532 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:45.791544 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:45.791595 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:45.791725 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:45.791726 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:45.791885 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:45.791941 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:45.791943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:45.792611 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:45.792612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:45.793150 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:45.793152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:45.793536 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:45.793537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:45.793597 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:45.794099 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:45.794104 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:45.795944 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":59,"threads_started":1}
W20260928 06:40:45.796152 28420 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:45.791478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:42829 (local address 127.18.211.126:34589)
0928 06:40:45.791589 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:45.791591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:45.791599 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:40:45.791638 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:45.791639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:45.791649 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:45.791721 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:45.792029 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:45.792534 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:45.793218 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:45.793451 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:45.793505 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:45.796042 (+ 2537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.65:42829: BlockingRecv error: recv got EOF from 127.18.211.65:42829 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
Sep 28 06:40:45 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:45 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:45.919891 28421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:45.915400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.66:36245)
0928 06:40:45.915534 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:45.915544 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:45.915601 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:45.915715 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:45.915716 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:45.915875 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:45.915924 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:45.915926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:45.916548 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:45.916549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:45.917087 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:45.917088 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:45.917415 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:45.917416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:45.917462 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:45.917920 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:45.917925 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:45.919832 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
W20260928 06:40:45.920024 28420 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:45.915450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:36245 (local address 127.18.211.126:34589)
0928 06:40:45.915500 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:45.915502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:45.915594 (+ 92us) server_negotiation.cc:407] Connection header received
0928 06:40:45.915655 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:45.915656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:45.915666 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:45.915702 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:45.915980 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:45.916477 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:45.917151 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:45.917349 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:45.917398 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:45.919956 (+ 2558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.66:36245: BlockingRecv error: recv got EOF from 127.18.211.66:36245 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 28 06:40:46 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:46 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:46.039233 28422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:46.034934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.67:40181)
0928 06:40:46.035068 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:46.035095 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:46.035142 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:46.035274 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:46.035275 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:46.035432 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:46.035490 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:46.035491 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:46.036098 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:46.036098 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:46.036612 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:46.036614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:46.036935 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:46.036935 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:46.036975 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:46.037398 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:46.037402 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:46.039169 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":57,"threads_started":1}
W20260928 06:40:46.039326 28420 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:46.035044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:40181 (local address 127.18.211.126:34589)
0928 06:40:46.035118 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:46.035120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:46.035150 (+ 30us) server_negotiation.cc:407] Connection header received
0928 06:40:46.035202 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:46.035202 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:46.035213 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:46.035251 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:46.035553 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:46.036040 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:46.036678 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:46.036876 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:46.036929 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:46.039244 (+ 2315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.67:40181: BlockingRecv error: recv got EOF from 127.18.211.67:40181 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 28 06:40:46 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:46 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:46.805611 28423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:46.796578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.65:35745)
0928 06:40:46.796722 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:46.796732 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:46.796780 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:46.796931 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:46.796932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:46.797091 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:46.797145 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:46.797147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:46.797900 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:46.797901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:46.798461 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:46.798463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:46.798834 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:46.798835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:46.798872 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:46.803711 (+ 4839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:46.803716 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:46.805530 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":67,"threads_started":1}
W20260928 06:40:46.805711 28424 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:46.796634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:35745 (local address 127.18.211.126:34589)
0928 06:40:46.796795 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:46.796797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:46.796808 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:40:46.796833 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:46.796833 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:46.796844 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:46.796924 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:46.797265 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:46.797824 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:46.798551 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:46.798770 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:46.798830 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:46.805642 (+ 6812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.65:35745: BlockingRecv error: recv got EOF from 127.18.211.65:35745 (error 108)
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":72,"threads_started":1}
Sep 28 06:40:46 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:46 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:46.924777 28425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:46.920351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.66:45259)
0928 06:40:46.920502 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:46.920515 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:46.920585 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:46.920713 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:46.920713 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:46.920886 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:46.920935 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:46.920937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:46.921559 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:46.921561 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:46.922113 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:46.922115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:46.922455 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:46.922456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:46.922502 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:46.922929 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:46.922933 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:46.924693 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":74,"threads_started":1}
W20260928 06:40:46.924840 28424 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:46.920431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:45259 (local address 127.18.211.126:34589)
0928 06:40:46.920486 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:46.920488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:46.920565 (+ 77us) server_negotiation.cc:407] Connection header received
0928 06:40:46.920645 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:46.920645 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:46.920657 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:46.920704 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:46.920990 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:46.921469 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:46.922192 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:46.922388 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:46.922438 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:46.924790 (+ 2352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.66:45259: BlockingRecv error: recv got EOF from 127.18.211.66:45259 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 28 06:40:47 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:47 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:47.044224 28426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:47.039783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.67:52273)
0928 06:40:47.039916 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:47.039928 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:47.039992 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:47.040130 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:47.040131 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:47.040309 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:47.040356 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:47.040358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:47.040996 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:47.040997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:47.041557 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:47.041559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:47.041882 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:47.041883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:47.041939 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:47.042396 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:47.042400 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:47.044136 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
W20260928 06:40:47.044291 28424 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:47.039854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:52273 (local address 127.18.211.126:34589)
0928 06:40:47.039904 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:47.039906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:47.039984 (+ 78us) server_negotiation.cc:407] Connection header received
0928 06:40:47.040050 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:47.040051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:47.040065 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:47.040112 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:47.040426 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:47.040925 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:47.041633 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:47.041823 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:47.041876 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:47.044236 (+ 2360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.67:52273: BlockingRecv error: recv got EOF from 127.18.211.67:52273 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 06:40:47 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:47 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:47.810788 28427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:47.806115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.65:57809)
0928 06:40:47.806299 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:47.806310 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:47.806364 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:47.806524 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:47.806525 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:47.806737 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:47.806795 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:47.806798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:47.807573 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:47.807574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:47.808132 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:47.808134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:47.808495 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:47.808495 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:47.808538 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:47.808981 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:47.808986 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:47.810727 (+ 1741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
W20260928 06:40:47.810925 28428 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:47.806226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:57809 (local address 127.18.211.126:34589)
0928 06:40:47.806362 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:47.806365 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:47.806403 (+ 38us) server_negotiation.cc:407] Connection header received
0928 06:40:47.806443 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:47.806444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:47.806454 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:47.806504 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:47.806873 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:47.807500 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:47.808220 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:47.808428 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:47.808479 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:47.810846 (+ 2367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.65:57809: BlockingRecv error: recv got EOF from 127.18.211.65:57809 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
Sep 28 06:40:47 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:47 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:47.929633 28429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:47.925217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.66:40051)
0928 06:40:47.925375 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:47.925386 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:47.925442 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:47.925543 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:47.925544 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:47.925702 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:47.925763 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:47.925765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:47.926511 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:47.926512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:47.927117 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:47.927120 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:47.927457 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:47.927457 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:47.927508 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:47.927914 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:47.927918 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:47.929565 (+ 1647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":92,"threads_started":1}
W20260928 06:40:47.929720 28428 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:47.925294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:40051 (local address 127.18.211.126:34589)
0928 06:40:47.925355 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:47.925358 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:47.925426 (+ 68us) server_negotiation.cc:407] Connection header received
0928 06:40:47.925484 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:47.925485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:47.925495 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:47.925537 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:47.925811 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:47.926444 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:47.927179 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:47.927395 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:47.927461 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:47.929661 (+ 2200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.66:40051: BlockingRecv error: recv got EOF from 127.18.211.66:40051 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 28 06:40:48 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:48 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:48.049357 28430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:48.044689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.67:49803)
0928 06:40:48.044839 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:48.044849 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:48.044904 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:48.045023 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:48.045023 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:48.045183 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:48.045231 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:48.045233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:48.045902 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:48.045903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:48.046458 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:48.046461 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:48.046802 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:48.046803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:48.046840 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:48.047336 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:48.047340 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:48.049248 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":67,"threads_started":1}
W20260928 06:40:48.049431 28428 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:48.044771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:49803 (local address 127.18.211.126:34589)
0928 06:40:48.044826 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:48.044829 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:48.044901 (+ 72us) server_negotiation.cc:407] Connection header received
0928 06:40:48.044945 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:48.044945 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:48.044957 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:48.045009 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:48.045326 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:48.045837 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:48.046539 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:48.046744 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:48.046798 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:48.049375 (+ 2577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.67:49803: BlockingRecv error: recv got EOF from 127.18.211.67:49803 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 28 06:40:48 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:48 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:48.821623 28431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:48.811299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.65:54523)
0928 06:40:48.811416 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:48.811427 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:48.811476 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:48.811710 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:48.811711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:48.811872 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:48.811919 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:48.811920 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:48.812574 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:48.812575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:48.813136 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:48.813139 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:48.813475 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:48.813476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:48.813523 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:48.819776 (+ 6253us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:48.819781 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:48.821552 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":66,"threads_started":1}
W20260928 06:40:48.821764 28432 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:48.811397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:54523 (local address 127.18.211.126:34589)
0928 06:40:48.811563 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:48.811565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:48.811574 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:40:48.811595 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:48.811596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:48.811605 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:48.811679 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:48.811982 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:48.812485 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:48.813217 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:48.813416 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:48.813467 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:48.821691 (+ 8224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.65:54523: BlockingRecv error: recv got EOF from 127.18.211.65:54523 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":95,"threads_started":1}
Sep 28 06:40:48 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:48 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:48.934283 28433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:48.930074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.66:35763)
0928 06:40:48.930198 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:48.930208 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:48.930255 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:48.930369 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:48.930369 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:48.930533 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:48.930582 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:48.930583 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:48.931243 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:48.931244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:48.931765 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:48.931767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:48.932079 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:48.932079 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:48.932123 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:48.932531 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:48.932535 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:48.934225 (+ 1690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":58,"threads_started":1}
W20260928 06:40:48.934378 28432 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:48.930197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:35763 (local address 127.18.211.126:34589)
0928 06:40:48.930254 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:48.930263 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:48.930272 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:40:48.930296 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:48.930297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:48.930307 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:48.930359 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:48.930635 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:48.931187 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:48.931822 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:48.932018 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:48.932067 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:48.934319 (+ 2252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.66:35763: BlockingRecv error: recv got EOF from 127.18.211.66:35763 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 28 06:40:49 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:49 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:49.054468 28434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:49.049873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.67:54929)
0928 06:40:49.050006 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:49.050018 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:49.050071 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:49.050280 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:49.050281 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:49.050474 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:49.050525 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:49.050527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:49.051310 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:49.051312 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:49.051851 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:49.051853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:49.052174 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:49.052175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:49.052214 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:49.052687 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:49.052691 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:49.054410 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":69,"threads_started":1}
W20260928 06:40:49.054564 28432 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:49.049979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:54929 (local address 127.18.211.126:34589)
0928 06:40:49.050053 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:49.050055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:49.050066 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:40:49.050132 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:49.050132 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:49.050146 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:49.050188 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:49.050618 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:49.051165 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:49.051926 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:49.052112 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:49.052164 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:49.054502 (+ 2338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.67:54929: BlockingRecv error: recv got EOF from 127.18.211.67:54929 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 06:40:49 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:49 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:49.826586 28435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:49.822115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.65:48513)
0928 06:40:49.822232 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:49.822243 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:49.822295 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:49.822440 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:49.822441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:49.822613 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:49.822660 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:49.822662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:49.823397 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:49.823398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:49.823943 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:49.823946 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:49.824321 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:49.824322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:49.824375 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:49.824804 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:49.824809 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:49.826526 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":64,"threads_started":1}
W20260928 06:40:49.826727 28436 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:49.822158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:48513 (local address 127.18.211.126:34589)
0928 06:40:49.822283 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:49.822286 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:49.822295 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:40:49.822347 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:49.822347 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:49.822361 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:49.822423 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:49.822726 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:49.823318 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:49.824018 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:49.824211 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:49.824291 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:49.826632 (+ 2341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.65:48513: BlockingRecv error: recv got EOF from 127.18.211.65:48513 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
Sep 28 06:40:49 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:49 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:49.939160 28437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:49.934724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.66:59629)
0928 06:40:49.934873 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:49.934883 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:49.934943 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:49.935131 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:49.935132 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:49.935310 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:49.935360 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:49.935362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:49.935996 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:49.935997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:49.936537 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:49.936538 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:49.936854 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:49.936854 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:49.936892 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:49.937294 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:49.937300 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:49.939064 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":77,"threads_started":1}
W20260928 06:40:49.939255 28436 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:49.934815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:59629 (local address 127.18.211.126:34589)
0928 06:40:49.934870 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:49.934872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:49.934936 (+ 64us) server_negotiation.cc:407] Connection header received
0928 06:40:49.935003 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:49.935004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:49.935015 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:49.935054 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:49.935422 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:49.935929 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:49.936600 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:49.936801 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:49.936849 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:49.939187 (+ 2338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.66:59629: BlockingRecv error: recv got EOF from 127.18.211.66:59629 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 28 06:40:50 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:50 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.67@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:50.059381 28438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:50.054923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.67:36617)
0928 06:40:50.055072 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:50.055102 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:50.055171 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:50.055373 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:50.055374 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:50.055548 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:50.055594 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:50.055595 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:50.056238 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:50.056239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:50.056784 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:50.056786 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:50.057145 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:50.057146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:50.057202 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:50.057641 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:50.057647 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:50.059322 (+ 1675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":85,"threads_started":1}
W20260928 06:40:50.059466 28436 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:50.055011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:36617 (local address 127.18.211.126:34589)
0928 06:40:50.055062 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:50.055065 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:50.055164 (+ 99us) server_negotiation.cc:407] Connection header received
0928 06:40:50.055255 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:50.055256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:50.055308 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:50.055356 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:50.055664 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:50.056169 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:50.056861 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:50.057083 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:50.057136 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:50.059405 (+ 2269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.67:36617: BlockingRecv error: recv got EOF from 127.18.211.67:36617 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 28 06:40:50 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:50 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.65@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:50.831920 28439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:50.827138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.65:36079)
0928 06:40:50.827280 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:50.827292 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:50.827348 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:50.827520 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:50.827521 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:50.827676 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:50.827728 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:50.827729 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:50.828437 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:50.828438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:50.828990 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:50.828992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:50.829384 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:50.829386 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:50.829473 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:50.829991 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:50.829995 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:50.831843 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":63,"threads_started":1}
W20260928 06:40:50.832028 28440 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:50.827187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:36079 (local address 127.18.211.126:34589)
0928 06:40:50.827337 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:50.827339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:50.827348 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:40:50.827405 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:50.827406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:50.827416 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:50.827462 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:50.827797 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:50.828360 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:50.829064 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:50.829258 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:50.829343 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:50.831954 (+ 2611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.65:36079: BlockingRecv error: recv got EOF from 127.18.211.65:36079 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":67,"threads_started":1}
Sep 28 06:40:50 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 06:40:50 dist-test-slave-b28r krb5kdc[27381](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.211.66@KRBTEST.COM for oryx/127.18.211.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 06:40:50.944732 28441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:50.939694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34589 (local address 127.18.211.66:59857)
0928 06:40:50.939886 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:50.939901 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:50.939982 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:50.940132 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:50.940133 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:50.940341 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:50.940391 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:50.940393 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:50.941183 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:50.941184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:50.941848 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:50.941853 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:50.942280 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:50.942281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:50.942332 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:50.942805 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:50.942811 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:50.944661 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.211.126:34589: Server oryx/127.18.211.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":90,"threads_started":1}
W20260928 06:40:50.944844 28440 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:50.939810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:59857 (local address 127.18.211.126:34589)
0928 06:40:50.939899 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:50.939901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:50.939971 (+ 70us) server_negotiation.cc:407] Connection header received
0928 06:40:50.940057 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:50.940059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:50.940070 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:50.940115 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:50.940471 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:50.941092 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:50.941955 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:50.942151 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:50.942214 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:50.944760 (+ 2546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.211.66:59857: BlockingRecv error: recv got EOF from 127.18.211.66:59857 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 06:40:50.983949 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 27964
I20260928 06:40:50.988080 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 28116
I20260928 06:40:50.991871 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 28248
I20260928 06:40:50.995522 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 27894
2026-09-28T06:40:50Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17272 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T06:40:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:40:51Z Disabled control of system clock
I20260928 06:40:51.021077 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:45433
--webserver_interface=127.18.211.126
--webserver_port=0
--builtin_ntp_servers=127.18.211.84:37555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:45433
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 06:40:51.100489 28445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:51.100723 28445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:51.100750 28445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:51.101924 28445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:40:51.101964 28445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:51.101979 28445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:51.101994 28445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:40:51.102010 28445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:40:51.103564 28445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:37555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:45433
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:45433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:51.103942 28445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:51.104233 28445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:51.109282 28451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:51.109333 28450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:51.109434 28445 server_base.cc:1034] running on GCE node
W20260928 06:40:51.109333 28453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:51.109753 28445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:51.110045 28445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:51.111148 28445 hybrid_clock.cc:648] HybridClock initialized: now 1790577651111132 us; error 26 us; skew 500 ppm
I20260928 06:40:51.112350 28445 webserver.cc:460] Webserver started at http://127.18.211.126:33605/ using document root <none> and password file <none>
I20260928 06:40:51.112577 28445 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:51.112639 28445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:51.112782 28445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:51.113888 28445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "b47a6503df604129bab4bca502d0f592"
format_stamp: "Formatted at 2026-09-28 06:40:51 on dist-test-slave-b28r"
server_key: "432bb865f6b78d78340598381c0b714b"
server_key_iv: "bfc466bfe58114ab52e26344ba65ff2a"
server_key_version: "encryptionkey@0"
I20260928 06:40:51.114208 28445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "b47a6503df604129bab4bca502d0f592"
format_stamp: "Formatted at 2026-09-28 06:40:51 on dist-test-slave-b28r"
server_key: "432bb865f6b78d78340598381c0b714b"
server_key_iv: "bfc466bfe58114ab52e26344ba65ff2a"
server_key_version: "encryptionkey@0"
I20260928 06:40:51.115518 28445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:40:51.116209 28459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:51.116433 28445 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:51.116525 28445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "b47a6503df604129bab4bca502d0f592"
format_stamp: "Formatted at 2026-09-28 06:40:51 on dist-test-slave-b28r"
server_key: "432bb865f6b78d78340598381c0b714b"
server_key_iv: "bfc466bfe58114ab52e26344ba65ff2a"
server_key_version: "encryptionkey@0"
I20260928 06:40:51.116590 28445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:51.128687 28445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:51.129012 28445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:51.129150 28445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:51.133041 28445 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:45433
I20260928 06:40:51.133075 28511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:45433 every 8 connection(s)
I20260928 06:40:51.133414 28445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:40:51.134089 28512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:51.135069 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 28445
I20260928 06:40:51.135218 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:40:51.135396 19277 external_mini_cluster.cc:1428] Setting key 6901924fdc9da7521e2fb21236215b61
I20260928 06:40:51.137323 28512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b47a6503df604129bab4bca502d0f592: Bootstrap starting.
I20260928 06:40:51.138032 28512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b47a6503df604129bab4bca502d0f592: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:51.138325 28512 log.cc:824] T 00000000000000000000000000000000 P b47a6503df604129bab4bca502d0f592: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:51.139056 28512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b47a6503df604129bab4bca502d0f592: No bootstrap required, opened a new log
I20260928 06:40:51.140980 28515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:51.135930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59646 (local address 127.18.211.126:45433)
0928 06:40:51.136117 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:51.136125 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:51.136151 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:40:51.136576 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:51.136584 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:51.136646 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:51.136778 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:51.137008 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:51.138274 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:51.139255 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:51.139664 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:51.139758 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:51.140425 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:51.140428 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:51.140429 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:40:51.140586 (+ 157us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:51.140619 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:40:51.140630 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:51.140677 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:51.140769 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 06:40:51.140830 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
I20260928 06:40:51.144358 28512 raft_consensus.cc:348] T 00000000000000000000000000000000 P b47a6503df604129bab4bca502d0f592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47a6503df604129bab4bca502d0f592" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 45433 } }
I20260928 06:40:51.144500 28512 raft_consensus.cc:374] T 00000000000000000000000000000000 P b47a6503df604129bab4bca502d0f592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:51.144521 28512 raft_consensus.cc:729] T 00000000000000000000000000000000 P b47a6503df604129bab4bca502d0f592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b47a6503df604129bab4bca502d0f592, State: Initialized, Role: FOLLOWER
I20260928 06:40:51.144627 28512 consensus_queue.cc:260] T 00000000000000000000000000000000 P b47a6503df604129bab4bca502d0f592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47a6503df604129bab4bca502d0f592" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 45433 } }
I20260928 06:40:51.144695 28512 raft_consensus.cc:388] T 00000000000000000000000000000000 P b47a6503df604129bab4bca502d0f592 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:40:51.144728 28512 raft_consensus.cc:482] T 00000000000000000000000000000000 P b47a6503df604129bab4bca502d0f592 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:40:51.144785 28512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b47a6503df604129bab4bca502d0f592 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:51.145347 28512 raft_consensus.cc:504] T 00000000000000000000000000000000 P b47a6503df604129bab4bca502d0f592 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47a6503df604129bab4bca502d0f592" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 45433 } }
I20260928 06:40:51.145438 28512 leader_election.cc:302] T 00000000000000000000000000000000 P b47a6503df604129bab4bca502d0f592 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b47a6503df604129bab4bca502d0f592; no voters:
I20260928 06:40:51.145605 28512 leader_election.cc:288] T 00000000000000000000000000000000 P b47a6503df604129bab4bca502d0f592 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:40:51.145684 28517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b47a6503df604129bab4bca502d0f592 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:51.145848 28517 raft_consensus.cc:686] T 00000000000000000000000000000000 P b47a6503df604129bab4bca502d0f592 [term 1 LEADER]: Becoming Leader. State: Replica: b47a6503df604129bab4bca502d0f592, State: Running, Role: LEADER
I20260928 06:40:51.145881 28512 sys_catalog.cc:564] T 00000000000000000000000000000000 P b47a6503df604129bab4bca502d0f592 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:40:51.145965 28517 consensus_queue.cc:237] T 00000000000000000000000000000000 P b47a6503df604129bab4bca502d0f592 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47a6503df604129bab4bca502d0f592" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 45433 } }
I20260928 06:40:51.146379 28519 sys_catalog.cc:455] T 00000000000000000000000000000000 P b47a6503df604129bab4bca502d0f592 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b47a6503df604129bab4bca502d0f592. Latest consensus state: current_term: 1 leader_uuid: "b47a6503df604129bab4bca502d0f592" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47a6503df604129bab4bca502d0f592" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 45433 } } }
I20260928 06:40:51.146551 28519 sys_catalog.cc:458] T 00000000000000000000000000000000 P b47a6503df604129bab4bca502d0f592 [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:51.146497 28518 sys_catalog.cc:455] T 00000000000000000000000000000000 P b47a6503df604129bab4bca502d0f592 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b47a6503df604129bab4bca502d0f592" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b47a6503df604129bab4bca502d0f592" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 45433 } } }
I20260928 06:40:51.146800 28526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:40:51.146871 28518 sys_catalog.cc:458] T 00000000000000000000000000000000 P b47a6503df604129bab4bca502d0f592 [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:51.147382 28526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:40:51.148921 28526 catalog_manager.cc:1347] Generated new cluster ID: 63dddd00510d4e1cadbfd95bb8ec7dc6
I20260928 06:40:51.148967 28526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:40:51.174660 28526 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:40:51.175454 28526 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:40:51.183907 28526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b47a6503df604129bab4bca502d0f592: Generated new TSK 0
I20260928 06:40:51.184122 28526 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:40:51.192955 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.65:0
--local_ip_for_outbound_sockets=127.18.211.65
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:45433
--builtin_ntp_servers=127.18.211.84:37555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 06:40:51.276612 28536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:51.276823 28536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:51.276854 28536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:51.278064 28536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:51.278102 28536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:51.278136 28536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.65
I20260928 06:40:51.279546 28536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:37555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:45433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.65
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:51.279798 28536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:51.280087 28536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:51.284812 28544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:51.284971 28542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:51.285248 28541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:51.285436 28536 server_base.cc:1034] running on GCE node
I20260928 06:40:51.285660 28536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:51.285898 28536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:51.287053 28536 hybrid_clock.cc:648] HybridClock initialized: now 1790577651287036 us; error 36 us; skew 500 ppm
I20260928 06:40:51.288429 28536 webserver.cc:460] Webserver started at http://127.18.211.65:42735/ using document root <none> and password file <none>
I20260928 06:40:51.288655 28536 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:51.288705 28536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:51.288866 28536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:51.289954 28536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/data/instance:
uuid: "6bc73d94119d4f139b964e12607912db"
format_stamp: "Formatted at 2026-09-28 06:40:51 on dist-test-slave-b28r"
server_key: "07c94d269e042fd42c9f88f51f3e6cb8"
server_key_iv: "153c1af1cd5624047b959f2fcd98bc52"
server_key_version: "encryptionkey@0"
I20260928 06:40:51.290292 28536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance:
uuid: "6bc73d94119d4f139b964e12607912db"
format_stamp: "Formatted at 2026-09-28 06:40:51 on dist-test-slave-b28r"
server_key: "07c94d269e042fd42c9f88f51f3e6cb8"
server_key_iv: "153c1af1cd5624047b959f2fcd98bc52"
server_key_version: "encryptionkey@0"
I20260928 06:40:51.291607 28536 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:40:51.292343 28550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:51.292495 28536 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:40:51.292590 28536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/wal
uuid: "6bc73d94119d4f139b964e12607912db"
format_stamp: "Formatted at 2026-09-28 06:40:51 on dist-test-slave-b28r"
server_key: "07c94d269e042fd42c9f88f51f3e6cb8"
server_key_iv: "153c1af1cd5624047b959f2fcd98bc52"
server_key_version: "encryptionkey@0"
I20260928 06:40:51.292660 28536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:51.310375 28536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:51.310678 28536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:51.310819 28536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:51.311054 28536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:51.311445 28536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:51.311483 28536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:51.311514 28536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:51.311553 28536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:51.318146 28536 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.65:34987
I20260928 06:40:51.318249 28663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.65:34987 every 8 connection(s)
I20260928 06:40:51.318594 28536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
I20260928 06:40:51.324596 28515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:51.319339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:41211 (local address 127.18.211.126:45433)
0928 06:40:51.319415 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:51.319417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:51.319670 (+ 253us) server_negotiation.cc:407] Connection header received
0928 06:40:51.320149 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:51.320150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:51.320161 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:51.320261 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:51.321225 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:51.321714 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:51.323210 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:51.323412 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:51.323463 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:51.324366 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:51.324367 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:51.324367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:40:51.324371 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:51.324385 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:40:51.324389 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:51.324430 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:51.324517 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 06:40:51.324539 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 06:40:51.324862 28666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:51.319357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:45433 (local address 127.18.211.65:41211)
0928 06:40:51.319549 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:51.319590 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:51.320064 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:51.320433 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:51.320438 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:51.320450 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:51.321121 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:51.321128 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:51.321808 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:51.321809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:51.323123 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:51.323126 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:51.323488 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:51.323489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:51.323648 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:51.324197 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:40:51.324197 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:51.324240 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:40:51.324270 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:51.324442 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:51.324446 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:51.324450 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:40:51.324511 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 06:40:51.324620 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
I20260928 06:40:51.325392 28664 heartbeater.cc:347] Connected to a master server at 127.18.211.126:45433
I20260928 06:40:51.325486 28664 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:51.325680 28664 heartbeater.cc:511] Master 127.18.211.126:45433 requested a full tablet report, sending...
I20260928 06:40:51.326247 28476 ts_manager.cc:194] Registered new tserver with Master: 6bc73d94119d4f139b964e12607912db (127.18.211.65:34987)
I20260928 06:40:51.326859 28476 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.211.65:41211
I20260928 06:40:51.328421 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 28536
I20260928 06:40:51.328492 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance
I20260928 06:40:51.328667 19277 external_mini_cluster.cc:1428] Setting key 2de3670cb42e05fe06b5a2df35144692
I20260928 06:40:51.329638 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.66:0
--local_ip_for_outbound_sockets=127.18.211.66
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:45433
--builtin_ntp_servers=127.18.211.84:37555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 06:40:51.407487 28667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:51.407671 28667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:51.407691 28667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:51.408797 28667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:51.408835 28667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:51.408859 28667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.66
I20260928 06:40:51.410121 28667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:37555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:45433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.66
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:51.410336 28667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:51.410605 28667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:51.415269 28672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:51.415282 28673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:51.415520 28667 server_base.cc:1034] running on GCE node
W20260928 06:40:51.415293 28675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:51.415727 28667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:51.415939 28667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:51.417095 28667 hybrid_clock.cc:648] HybridClock initialized: now 1790577651417084 us; error 32 us; skew 500 ppm
I20260928 06:40:51.418332 28667 webserver.cc:460] Webserver started at http://127.18.211.66:43465/ using document root <none> and password file <none>
I20260928 06:40:51.418541 28667 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:51.418589 28667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:51.418732 28667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:51.419889 28667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/data/instance:
uuid: "c346cfd3f3db464dad3769339f763493"
format_stamp: "Formatted at 2026-09-28 06:40:51 on dist-test-slave-b28r"
server_key: "7ad315b9f89f3393e6d59457b8cc6979"
server_key_iv: "f4bf4ae131a8353f6e807aff86d10ec2"
server_key_version: "encryptionkey@0"
I20260928 06:40:51.420217 28667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance:
uuid: "c346cfd3f3db464dad3769339f763493"
format_stamp: "Formatted at 2026-09-28 06:40:51 on dist-test-slave-b28r"
server_key: "7ad315b9f89f3393e6d59457b8cc6979"
server_key_iv: "f4bf4ae131a8353f6e807aff86d10ec2"
server_key_version: "encryptionkey@0"
I20260928 06:40:51.421502 28667 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:40:51.422241 28681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:51.422397 28667 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:40:51.422483 28667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/wal
uuid: "c346cfd3f3db464dad3769339f763493"
format_stamp: "Formatted at 2026-09-28 06:40:51 on dist-test-slave-b28r"
server_key: "7ad315b9f89f3393e6d59457b8cc6979"
server_key_iv: "f4bf4ae131a8353f6e807aff86d10ec2"
server_key_version: "encryptionkey@0"
I20260928 06:40:51.422552 28667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:51.459831 28667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:51.460163 28667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:51.460311 28667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:51.460561 28667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:51.460917 28667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:51.460951 28667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:51.460980 28667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:51.460997 28667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:51.466316 28667 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.66:39151
I20260928 06:40:51.466380 28794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.66:39151 every 8 connection(s)
I20260928 06:40:51.466672 28667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
I20260928 06:40:51.471388 28515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:51.467370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:37505 (local address 127.18.211.126:45433)
0928 06:40:51.467436 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:51.467438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:51.467658 (+ 220us) server_negotiation.cc:407] Connection header received
0928 06:40:51.468138 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:51.468139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:51.468149 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:51.468204 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:51.468803 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:51.469284 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:51.470230 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:51.470408 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:51.470454 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:51.471179 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:51.471180 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:51.471181 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:40:51.471185 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:51.471199 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:40:51.471203 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:51.471239 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:51.471301 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 06:40:51.471328 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:40:51.471599 28797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:51.467354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:45433 (local address 127.18.211.66:37505)
0928 06:40:51.467539 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:51.467573 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:51.468059 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:51.468259 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:51.468262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:51.468271 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:51.468736 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:51.468740 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:51.469383 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:51.469384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:51.470156 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:51.470158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:51.470467 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:51.470468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:51.470571 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:51.471021 (+ 450us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:40:51.471021 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:51.471048 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:40:51.471071 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:51.471246 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:51.471249 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:51.471251 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:40:51.471299 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 06:40:51.471402 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
I20260928 06:40:51.472013 28795 heartbeater.cc:347] Connected to a master server at 127.18.211.126:45433
I20260928 06:40:51.472105 28795 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:51.472283 28795 heartbeater.cc:511] Master 127.18.211.126:45433 requested a full tablet report, sending...
I20260928 06:40:51.472620 28476 ts_manager.cc:194] Registered new tserver with Master: c346cfd3f3db464dad3769339f763493 (127.18.211.66:39151)
I20260928 06:40:51.472963 28476 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.211.66:37505
I20260928 06:40:51.474408 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 28667
I20260928 06:40:51.474494 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance
I20260928 06:40:51.474684 19277 external_mini_cluster.cc:1428] Setting key 50f93f93d2b519b9ccffbe7d92e64353
I20260928 06:40:51.475574 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.67:0
--local_ip_for_outbound_sockets=127.18.211.67
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:45433
--builtin_ntp_servers=127.18.211.84:37555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 06:40:51.556023 28798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:51.556236 28798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:51.556264 28798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:51.557416 28798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:51.557458 28798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:51.557489 28798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.67
I20260928 06:40:51.558805 28798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:37555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:45433
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.67
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:51.559033 28798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:51.559378 28798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:51.564054 28803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:51.564041 28806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:51.564152 28798 server_base.cc:1034] running on GCE node
W20260928 06:40:51.564041 28804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:51.564525 28798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:51.564762 28798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:51.565904 28798 hybrid_clock.cc:648] HybridClock initialized: now 1790577651565891 us; error 28 us; skew 500 ppm
I20260928 06:40:51.567107 28798 webserver.cc:460] Webserver started at http://127.18.211.67:40641/ using document root <none> and password file <none>
I20260928 06:40:51.567330 28798 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:51.567381 28798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:51.567502 28798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:51.568604 28798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/data/instance:
uuid: "6d565ebc7d834942b08cb8b5f16db9a7"
format_stamp: "Formatted at 2026-09-28 06:40:51 on dist-test-slave-b28r"
server_key: "32ff9a7fe56a6c723b4c413ff94bcfc8"
server_key_iv: "f0319ece9e7b149f69a904b52fd925d0"
server_key_version: "encryptionkey@0"
I20260928 06:40:51.568936 28798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance:
uuid: "6d565ebc7d834942b08cb8b5f16db9a7"
format_stamp: "Formatted at 2026-09-28 06:40:51 on dist-test-slave-b28r"
server_key: "32ff9a7fe56a6c723b4c413ff94bcfc8"
server_key_iv: "f0319ece9e7b149f69a904b52fd925d0"
server_key_version: "encryptionkey@0"
I20260928 06:40:51.570135 28798 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:40:51.570792 28812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:51.570940 28798 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:40:51.571013 28798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/wal
uuid: "6d565ebc7d834942b08cb8b5f16db9a7"
format_stamp: "Formatted at 2026-09-28 06:40:51 on dist-test-slave-b28r"
server_key: "32ff9a7fe56a6c723b4c413ff94bcfc8"
server_key_iv: "f0319ece9e7b149f69a904b52fd925d0"
server_key_version: "encryptionkey@0"
I20260928 06:40:51.571075 28798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:51.577597 28798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:51.577857 28798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:51.577975 28798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:51.578166 28798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:51.578466 28798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:51.578498 28798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:51.578524 28798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:51.578549 28798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:51.583943 28798 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.67:44893
I20260928 06:40:51.583992 28925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.67:44893 every 8 connection(s)
I20260928 06:40:51.584285 28798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
I20260928 06:40:51.589228 28515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:51.584985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:34741 (local address 127.18.211.126:45433)
0928 06:40:51.585035 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:51.585037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:51.585253 (+ 216us) server_negotiation.cc:407] Connection header received
0928 06:40:51.585755 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:51.585757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:51.585768 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:51.585809 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:51.586487 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:51.586979 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:51.588017 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:51.588203 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:51.588261 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:51.588980 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:51.588982 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:51.588982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:40:51.588987 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:51.589002 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:40:51.589006 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:51.589048 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:51.589126 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 06:40:51.589151 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:40:51.589411 28928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:51.584929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:45433 (local address 127.18.211.67:34741)
0928 06:40:51.585140 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:51.585173 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:51.585662 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:51.585887 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:51.585891 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:51.585902 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:51.586399 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:51.586405 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:51.587064 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:51.587065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:51.587931 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:51.587934 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:51.588273 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:51.588274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:51.588406 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:51.588853 (+ 447us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:40:51.588853 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:51.588879 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:40:51.588901 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:51.589053 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:51.589058 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:51.589061 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:40:51.589123 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 06:40:51.589221 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":81,"threads_started":1}
I20260928 06:40:51.589828 28926 heartbeater.cc:347] Connected to a master server at 127.18.211.126:45433
I20260928 06:40:51.589918 28926 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:51.589984 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 28798
I20260928 06:40:51.590066 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance
I20260928 06:40:51.590099 28926 heartbeater.cc:511] Master 127.18.211.126:45433 requested a full tablet report, sending...
I20260928 06:40:51.590281 19277 external_mini_cluster.cc:1428] Setting key 18d5b055cf40465811666b15d361e5e2
I20260928 06:40:51.590507 28476 ts_manager.cc:194] Registered new tserver with Master: 6d565ebc7d834942b08cb8b5f16db9a7 (127.18.211.67:44893)
I20260928 06:40:51.590971 28476 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.211.67:34741
I20260928 06:40:51.591109 19277 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 06:40:51.592720 28515 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:51.592332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59652 (local address 127.18.211.126:45433)
0928 06:40:51.592365 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:51.592367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:51.592486 (+ 119us) server_negotiation.cc:407] Connection header received
0928 06:40:51.592522 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:51.592523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:51.592531 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:51.592574 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:51.592665 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59652: BlockingRecv error: recv got EOF from 127.0.0.1:59652 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:40:51.592823 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 28536
I20260928 06:40:51.597892 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 28667
I20260928 06:40:51.601325 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 28798
I20260928 06:40:51.604952 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 28445
2026-09-28T06:40:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (608 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T06:40:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:40:51Z Disabled control of system clock
I20260928 06:40:51.629860 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:39003
--webserver_interface=127.18.211.126
--webserver_port=0
--builtin_ntp_servers=127.18.211.84:38193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:39003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 06:40:51.711352 28937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:51.711555 28937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:51.711582 28937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:51.712678 28937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:40:51.712716 28937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:51.712733 28937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:51.712749 28937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:40:51.712773 28937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:40:51.714080 28937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:38193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:39003
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:39003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:51.714313 28937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:51.714566 28937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:51.719128 28942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:51.719266 28943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:51.719138 28945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:51.719372 28937 server_base.cc:1034] running on GCE node
I20260928 06:40:51.719676 28937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:51.719980 28937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:51.721123 28937 hybrid_clock.cc:648] HybridClock initialized: now 1790577651721105 us; error 37 us; skew 500 ppm
I20260928 06:40:51.722326 28937 webserver.cc:460] Webserver started at http://127.18.211.126:42363/ using document root <none> and password file <none>
I20260928 06:40:51.722527 28937 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:51.722574 28937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:51.722697 28937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:51.723800 28937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "51846fb2b17143cbae420f1fae594d92"
format_stamp: "Formatted at 2026-09-28 06:40:51 on dist-test-slave-b28r"
server_key: "21ad4510d7c0924e7e31f17ce85c9172"
server_key_iv: "6bd16083bf72ed3f4938b5930df36900"
server_key_version: "encryptionkey@0"
I20260928 06:40:51.724118 28937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "51846fb2b17143cbae420f1fae594d92"
format_stamp: "Formatted at 2026-09-28 06:40:51 on dist-test-slave-b28r"
server_key: "21ad4510d7c0924e7e31f17ce85c9172"
server_key_iv: "6bd16083bf72ed3f4938b5930df36900"
server_key_version: "encryptionkey@0"
I20260928 06:40:51.725306 28937 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:40:51.725991 28951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:51.726152 28937 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:40:51.726233 28937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "51846fb2b17143cbae420f1fae594d92"
format_stamp: "Formatted at 2026-09-28 06:40:51 on dist-test-slave-b28r"
server_key: "21ad4510d7c0924e7e31f17ce85c9172"
server_key_iv: "6bd16083bf72ed3f4938b5930df36900"
server_key_version: "encryptionkey@0"
I20260928 06:40:51.726300 28937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:51.729494 28937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:51.729722 28937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:51.729836 28937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:51.733405 28937 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:39003
I20260928 06:40:51.733434 29003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:39003 every 8 connection(s)
I20260928 06:40:51.733721 28937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:40:51.734316 29004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:51.736917 29004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51846fb2b17143cbae420f1fae594d92: Bootstrap starting.
I20260928 06:40:51.737445 29004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51846fb2b17143cbae420f1fae594d92: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:51.737701 29004 log.cc:824] T 00000000000000000000000000000000 P 51846fb2b17143cbae420f1fae594d92: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:51.738327 29004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51846fb2b17143cbae420f1fae594d92: No bootstrap required, opened a new log
I20260928 06:40:51.739899 29004 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51846fb2b17143cbae420f1fae594d92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51846fb2b17143cbae420f1fae594d92" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 39003 } }
I20260928 06:40:51.740028 29004 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51846fb2b17143cbae420f1fae594d92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:51.740049 29004 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51846fb2b17143cbae420f1fae594d92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51846fb2b17143cbae420f1fae594d92, State: Initialized, Role: FOLLOWER
I20260928 06:40:51.740197 29004 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51846fb2b17143cbae420f1fae594d92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51846fb2b17143cbae420f1fae594d92" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 39003 } }
I20260928 06:40:51.740258 29004 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51846fb2b17143cbae420f1fae594d92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:40:51.740278 29004 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51846fb2b17143cbae420f1fae594d92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:40:51.740305 29004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51846fb2b17143cbae420f1fae594d92 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:51.740792 29004 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51846fb2b17143cbae420f1fae594d92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51846fb2b17143cbae420f1fae594d92" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 39003 } }
I20260928 06:40:51.740875 29004 leader_election.cc:302] T 00000000000000000000000000000000 P 51846fb2b17143cbae420f1fae594d92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51846fb2b17143cbae420f1fae594d92; no voters:
I20260928 06:40:51.741014 29004 leader_election.cc:288] T 00000000000000000000000000000000 P 51846fb2b17143cbae420f1fae594d92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:40:51.741096 29007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51846fb2b17143cbae420f1fae594d92 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:51.741257 29007 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51846fb2b17143cbae420f1fae594d92 [term 1 LEADER]: Becoming Leader. State: Replica: 51846fb2b17143cbae420f1fae594d92, State: Running, Role: LEADER
I20260928 06:40:51.741256 29004 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51846fb2b17143cbae420f1fae594d92 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:40:51.741417 29007 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51846fb2b17143cbae420f1fae594d92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51846fb2b17143cbae420f1fae594d92" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 39003 } }
I20260928 06:40:51.741773 29009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51846fb2b17143cbae420f1fae594d92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51846fb2b17143cbae420f1fae594d92. Latest consensus state: current_term: 1 leader_uuid: "51846fb2b17143cbae420f1fae594d92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51846fb2b17143cbae420f1fae594d92" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 39003 } } }
I20260928 06:40:51.741923 29009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51846fb2b17143cbae420f1fae594d92 [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:51.741992 29008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51846fb2b17143cbae420f1fae594d92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51846fb2b17143cbae420f1fae594d92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51846fb2b17143cbae420f1fae594d92" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 39003 } } }
I20260928 06:40:51.742141 29008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51846fb2b17143cbae420f1fae594d92 [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:51.742338 29016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:40:51.742792 29016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:40:51.743798 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 28937
I20260928 06:40:51.743896 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:40:51.744095 19277 external_mini_cluster.cc:1428] Setting key 0b876f3afdeab864541bdb56c276bb58
I20260928 06:40:51.744345 29016 catalog_manager.cc:1347] Generated new cluster ID: 0ad89f8a7f95406b81efcc2c69af09da
I20260928 06:40:51.744391 29016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:40:51.746286 29027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:51.744682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52308 (local address 127.18.211.126:39003)
0928 06:40:51.744861 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:51.744869 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:51.744897 (+ 28us) server_negotiation.cc:407] Connection header received
0928 06:40:51.745004 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:51.745010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:51.745059 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:51.745183 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:51.745797 (+ 614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:51.745800 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:51.745800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:40:51.745804 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:51.745845 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:40:51.745858 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:51.745914 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:51.745981 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 06:40:51.746111 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":92,"threads_started":1}
I20260928 06:40:51.749778 29016 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:40:51.749919 29016 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:40:51.757745 29016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51846fb2b17143cbae420f1fae594d92: Generated new TSK 0
I20260928 06:40:51.757922 29016 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:40:51.766314 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.65:0
--local_ip_for_outbound_sockets=127.18.211.65
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:39003
--builtin_ntp_servers=127.18.211.84:38193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 06:40:51.847744 29028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:51.847955 29028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:51.847976 29028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:51.849113 29028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:51.849154 29028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:51.849179 29028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.65
I20260928 06:40:51.850492 29028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:38193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:39003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.65
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:51.850700 29028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:51.850981 29028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:40:51.856678 29028 server_base.cc:1034] running on GCE node
W20260928 06:40:51.856678 29034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:51.856866 29033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:51.856662 29036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:51.857146 29028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:51.857446 29028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:51.858618 29028 hybrid_clock.cc:648] HybridClock initialized: now 1790577651858596 us; error 38 us; skew 500 ppm
I20260928 06:40:51.863451 29028 webserver.cc:460] Webserver started at http://127.18.211.65:43095/ using document root <none> and password file <none>
I20260928 06:40:51.863715 29028 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:51.863782 29028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:51.863907 29028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:51.865346 29028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/data/instance:
uuid: "a6d12c2330924e42ae94b5c240cfd6f3"
format_stamp: "Formatted at 2026-09-28 06:40:51 on dist-test-slave-b28r"
server_key: "169aaea24af1f58412836e5808efe4d6"
server_key_iv: "01c92828c67f378928e61f47ceef9876"
server_key_version: "encryptionkey@0"
I20260928 06:40:51.865721 29028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance:
uuid: "a6d12c2330924e42ae94b5c240cfd6f3"
format_stamp: "Formatted at 2026-09-28 06:40:51 on dist-test-slave-b28r"
server_key: "169aaea24af1f58412836e5808efe4d6"
server_key_iv: "01c92828c67f378928e61f47ceef9876"
server_key_version: "encryptionkey@0"
I20260928 06:40:51.867205 29028 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260928 06:40:51.868171 29042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:51.868362 29028 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:51.868455 29028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/wal
uuid: "a6d12c2330924e42ae94b5c240cfd6f3"
format_stamp: "Formatted at 2026-09-28 06:40:51 on dist-test-slave-b28r"
server_key: "169aaea24af1f58412836e5808efe4d6"
server_key_iv: "01c92828c67f378928e61f47ceef9876"
server_key_version: "encryptionkey@0"
I20260928 06:40:51.868512 29028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:51.871762 29028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:51.871978 29028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:51.872068 29028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:51.872224 29028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:51.872457 29028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:51.872486 29028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:51.872503 29028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:51.872520 29028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:51.877805 29028 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.65:41149
I20260928 06:40:51.877844 29155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.65:41149 every 8 connection(s)
I20260928 06:40:51.878129 29028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
I20260928 06:40:51.880874 29027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:51.878792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:56819 (local address 127.18.211.126:39003)
0928 06:40:51.878850 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:51.878852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:51.879049 (+ 197us) server_negotiation.cc:407] Connection header received
0928 06:40:51.879514 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:51.879515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:51.879522 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:51.879575 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:51.880673 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:51.880674 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:51.880674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:40:51.880678 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:51.880693 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:40:51.880697 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:51.880749 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:51.880812 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 06:40:51.880824 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 06:40:51.881037 29158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:51.878775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:39003 (local address 127.18.211.65:56819)
0928 06:40:51.878940 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:51.878972 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:51.879426 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:51.879643 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:51.879646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:51.879653 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:51.880533 (+ 880us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:40:51.880534 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:51.880561 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:40:51.880588 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:51.880750 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:51.880753 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:51.880755 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:40:51.880809 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 06:40:51.880854 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
I20260928 06:40:51.881544 29156 heartbeater.cc:347] Connected to a master server at 127.18.211.126:39003
I20260928 06:40:51.881623 29156 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:51.881757 29156 heartbeater.cc:511] Master 127.18.211.126:39003 requested a full tablet report, sending...
I20260928 06:40:51.882246 28968 ts_manager.cc:194] Registered new tserver with Master: a6d12c2330924e42ae94b5c240cfd6f3 (127.18.211.65:41149)
I20260928 06:40:51.883760 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 29028
I20260928 06:40:51.883852 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance
I20260928 06:40:51.884012 19277 external_mini_cluster.cc:1428] Setting key 3cb0848860dbdfae38a9447222c5cefc
I20260928 06:40:51.885051 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.66:0
--local_ip_for_outbound_sockets=127.18.211.66
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:39003
--builtin_ntp_servers=127.18.211.84:38193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 06:40:51.965570 29159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:51.965764 29159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:51.965785 29159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:51.966888 29159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:51.966921 29159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:51.966945 29159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.66
I20260928 06:40:51.968323 29159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:38193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:39003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.66
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:51.968528 29159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:51.968845 29159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:51.973625 29167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:51.973667 29159 server_base.cc:1034] running on GCE node
W20260928 06:40:51.973640 29164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:51.973634 29165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:51.974042 29159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:51.974251 29159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:51.975404 29159 hybrid_clock.cc:648] HybridClock initialized: now 1790577651975378 us; error 40 us; skew 500 ppm
I20260928 06:40:51.976583 29159 webserver.cc:460] Webserver started at http://127.18.211.66:40761/ using document root <none> and password file <none>
I20260928 06:40:51.976794 29159 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:51.976838 29159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:51.976969 29159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:51.977982 29159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/data/instance:
uuid: "b4e83b8e204c44129ceee3edd7a7a543"
format_stamp: "Formatted at 2026-09-28 06:40:51 on dist-test-slave-b28r"
server_key: "3004e25a9ddda3bf9e585b7e7437e4b2"
server_key_iv: "f6173feb528ca5df88b78c6a4e25e581"
server_key_version: "encryptionkey@0"
I20260928 06:40:51.978292 29159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance:
uuid: "b4e83b8e204c44129ceee3edd7a7a543"
format_stamp: "Formatted at 2026-09-28 06:40:51 on dist-test-slave-b28r"
server_key: "3004e25a9ddda3bf9e585b7e7437e4b2"
server_key_iv: "f6173feb528ca5df88b78c6a4e25e581"
server_key_version: "encryptionkey@0"
I20260928 06:40:51.979584 29159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:40:51.980253 29173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:51.980446 29159 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:40:51.980530 29159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/wal
uuid: "b4e83b8e204c44129ceee3edd7a7a543"
format_stamp: "Formatted at 2026-09-28 06:40:51 on dist-test-slave-b28r"
server_key: "3004e25a9ddda3bf9e585b7e7437e4b2"
server_key_iv: "f6173feb528ca5df88b78c6a4e25e581"
server_key_version: "encryptionkey@0"
I20260928 06:40:51.980598 29159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:51.983804 29159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:51.984036 29159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:51.984148 29159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:51.984330 29159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:51.984596 29159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:51.984627 29159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:51.984653 29159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:51.984676 29159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:51.989894 29159 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.66:41639
I20260928 06:40:51.989967 29286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.66:41639 every 8 connection(s)
I20260928 06:40:51.990216 29159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
I20260928 06:40:51.992959 29027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:51.990887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:58003 (local address 127.18.211.126:39003)
0928 06:40:51.990964 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:51.990966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:51.991203 (+ 237us) server_negotiation.cc:407] Connection header received
0928 06:40:51.991679 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:51.991680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:51.991686 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:51.991738 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:51.992780 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:51.992782 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:51.992782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:40:51.992786 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:51.992801 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:40:51.992806 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:51.992850 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:51.992895 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 06:40:51.992906 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:40:51.993119 29289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:51.990871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:39003 (local address 127.18.211.66:58003)
0928 06:40:51.991063 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:51.991130 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:51.991596 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:51.991790 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:51.991793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:51.991801 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:51.992644 (+ 843us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:40:51.992645 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:51.992672 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:40:51.992698 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:51.992849 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:51.992851 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:51.992854 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:40:51.992903 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 06:40:51.992946 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":102,"threads_started":1}
I20260928 06:40:51.993584 29287 heartbeater.cc:347] Connected to a master server at 127.18.211.126:39003
I20260928 06:40:51.993716 29287 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:51.993862 29287 heartbeater.cc:511] Master 127.18.211.126:39003 requested a full tablet report, sending...
I20260928 06:40:51.994179 28968 ts_manager.cc:194] Registered new tserver with Master: b4e83b8e204c44129ceee3edd7a7a543 (127.18.211.66:41639)
I20260928 06:40:51.999387 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 29159
I20260928 06:40:51.999465 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance
I20260928 06:40:51.999624 19277 external_mini_cluster.cc:1428] Setting key 1a2ec870b7f78995b47271545e1dce98
I20260928 06:40:52.000521 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.67:0
--local_ip_for_outbound_sockets=127.18.211.67
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:39003
--builtin_ntp_servers=127.18.211.84:38193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 06:40:52.080080 29290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:52.080299 29290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:52.080322 29290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:52.081439 29290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:52.081473 29290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:52.081496 29290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.67
I20260928 06:40:52.082765 29290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:38193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:39003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.67
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:52.083027 29290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:52.083328 29290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:52.088022 29298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:52.088042 29296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:52.088022 29295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:52.088104 29290 server_base.cc:1034] running on GCE node
I20260928 06:40:52.088454 29290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:52.088708 29290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:52.089879 29290 hybrid_clock.cc:648] HybridClock initialized: now 1790577652089863 us; error 31 us; skew 500 ppm
I20260928 06:40:52.091074 29290 webserver.cc:460] Webserver started at http://127.18.211.67:40813/ using document root <none> and password file <none>
I20260928 06:40:52.091351 29290 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:52.091399 29290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:52.091540 29290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:52.092564 29290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/data/instance:
uuid: "3fac9a93d6aa4a768f120e49449c23b1"
format_stamp: "Formatted at 2026-09-28 06:40:52 on dist-test-slave-b28r"
server_key: "9ec6f7c4f256588ff2cf6fb846548cf5"
server_key_iv: "ff4f2047226972b90003530953f3dbe3"
server_key_version: "encryptionkey@0"
I20260928 06:40:52.092900 29290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance:
uuid: "3fac9a93d6aa4a768f120e49449c23b1"
format_stamp: "Formatted at 2026-09-28 06:40:52 on dist-test-slave-b28r"
server_key: "9ec6f7c4f256588ff2cf6fb846548cf5"
server_key_iv: "ff4f2047226972b90003530953f3dbe3"
server_key_version: "encryptionkey@0"
I20260928 06:40:52.094110 29290 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:40:52.094913 29304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:52.095119 29290 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:52.095197 29290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/wal
uuid: "3fac9a93d6aa4a768f120e49449c23b1"
format_stamp: "Formatted at 2026-09-28 06:40:52 on dist-test-slave-b28r"
server_key: "9ec6f7c4f256588ff2cf6fb846548cf5"
server_key_iv: "ff4f2047226972b90003530953f3dbe3"
server_key_version: "encryptionkey@0"
I20260928 06:40:52.095295 29290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:52.098469 29290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:52.098677 29290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:52.098768 29290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:52.098913 29290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:52.099179 29290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:52.099221 29290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:52.099246 29290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:52.099262 29290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:52.104406 29290 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.67:33849
I20260928 06:40:52.104470 29417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.67:33849 every 8 connection(s)
I20260928 06:40:52.104758 29290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
I20260928 06:40:52.107551 29027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:52.105498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:46901 (local address 127.18.211.126:39003)
0928 06:40:52.105570 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:52.105572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:52.105770 (+ 198us) server_negotiation.cc:407] Connection header received
0928 06:40:52.106202 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:52.106203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:52.106208 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:52.106274 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:52.107357 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:52.107358 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:52.107359 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:40:52.107362 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:52.107376 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:40:52.107380 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:52.107428 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:52.107486 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 06:40:52.107497 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 06:40:52.107745 29420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:52.105440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:39003 (local address 127.18.211.67:46901)
0928 06:40:52.105644 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:52.105683 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:52.106128 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:52.106335 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:52.106338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:52.106346 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:52.107209 (+ 863us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:40:52.107209 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:52.107240 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:40:52.107275 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:52.107432 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:52.107435 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:52.107438 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:40:52.107490 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 06:40:52.107538 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":102,"threads_started":1}
I20260928 06:40:52.108178 29418 heartbeater.cc:347] Connected to a master server at 127.18.211.126:39003
I20260928 06:40:52.108264 29418 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:52.108378 29418 heartbeater.cc:511] Master 127.18.211.126:39003 requested a full tablet report, sending...
I20260928 06:40:52.108712 28968 ts_manager.cc:194] Registered new tserver with Master: 3fac9a93d6aa4a768f120e49449c23b1 (127.18.211.67:33849)
I20260928 06:40:52.114854 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 29290
I20260928 06:40:52.114934 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance
I20260928 06:40:52.115116 19277 external_mini_cluster.cc:1428] Setting key b4ecddeed87c72a5d8e545926c7ea6df
I20260928 06:40:52.116005 19277 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 06:40:52.117720 29027 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:40:52.117318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52324 (local address 127.18.211.126:39003)
0928 06:40:52.117367 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:52.117369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:52.117412 (+ 43us) server_negotiation.cc:407] Connection header received
0928 06:40:52.117493 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:52.117494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:52.117500 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:52.117545 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:52.117660 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52324: BlockingRecv error: recv got EOF from 127.0.0.1:52324 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:40:52.117811 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 29028
I20260928 06:40:52.121685 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 29159
I20260928 06:40:52.125937 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 29290
I20260928 06:40:52.129801 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 28937
2026-09-28T06:40:52Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (524 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:40:52 dist-test-slave-b28r krb5kdc[29428](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:40:52 dist-test-slave-b28r krb5kdc[29428](info): set up 2 sockets
Sep 28 06:40:52 dist-test-slave-b28r krb5kdc[29428](info): commencing operation
krb5kdc: starting...
W20260928 06:40:53.080696 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:40:53 dist-test-slave-b28r krb5kdc[29428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:40:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:40:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.211.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
I20260928 06:40:53.193610 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:42169
--webserver_interface=127.18.211.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:42169
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:53.273865 29441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:53.274077 29441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:53.274101 29441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:53.275230 29441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:40:53.275269 29441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:53.275285 29441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:53.275298 29441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:40:53.275317 29441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:40:53.276718 29441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:42169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:42169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:53.276913 29441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:53.277097 29441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:53.281767 29449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:53.281881 29441 server_base.cc:1034] running on GCE node
W20260928 06:40:53.281829 29446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:53.281800 29447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:53.282225 29441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:53.282500 29441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:53.283646 29441 hybrid_clock.cc:648] HybridClock initialized: now 1790577653283634 us; error 28 us; skew 500 ppm
Sep 28 06:40:53 dist-test-slave-b28r krb5kdc[29428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577653, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:53.285974 29441 init.cc:377] Logged in from keytab as kudu/127.18.211.126@KRBTEST.COM (short username kudu)
I20260928 06:40:53.286398 29441 webserver.cc:460] Webserver started at http://127.18.211.126:32837/ using document root <none> and password file <none>
I20260928 06:40:53.286605 29441 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:53.286652 29441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:53.286777 29441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:53.287846 29441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "4a64621b72c149d08a24e8050b9ccbe9"
format_stamp: "Formatted at 2026-09-28 06:40:53 on dist-test-slave-b28r"
server_key: "656010459d500dd7751d5384e54cbd78"
server_key_iv: "c15fd81146c4e8fb66824b9121679c34"
server_key_version: "encryptionkey@0"
I20260928 06:40:53.288174 29441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "4a64621b72c149d08a24e8050b9ccbe9"
format_stamp: "Formatted at 2026-09-28 06:40:53 on dist-test-slave-b28r"
server_key: "656010459d500dd7751d5384e54cbd78"
server_key_iv: "c15fd81146c4e8fb66824b9121679c34"
server_key_version: "encryptionkey@0"
I20260928 06:40:53.289580 29441 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:40:53.290277 29456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:53.290457 29441 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:40:53.290539 29441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "4a64621b72c149d08a24e8050b9ccbe9"
format_stamp: "Formatted at 2026-09-28 06:40:53 on dist-test-slave-b28r"
server_key: "656010459d500dd7751d5384e54cbd78"
server_key_iv: "c15fd81146c4e8fb66824b9121679c34"
server_key_version: "encryptionkey@0"
I20260928 06:40:53.290604 29441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:53.305195 29441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:53.307533 29441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:53.307662 29441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:53.311463 29441 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:42169
I20260928 06:40:53.311504 29508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:42169 every 8 connection(s)
I20260928 06:40:53.311843 29441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:40:53.312930 29509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:53.315142 29509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a64621b72c149d08a24e8050b9ccbe9: Bootstrap starting.
I20260928 06:40:53.315687 29509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a64621b72c149d08a24e8050b9ccbe9: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:53.315944 29509 log.cc:824] T 00000000000000000000000000000000 P 4a64621b72c149d08a24e8050b9ccbe9: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:53.316583 29509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a64621b72c149d08a24e8050b9ccbe9: No bootstrap required, opened a new log
I20260928 06:40:53.318097 29509 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a64621b72c149d08a24e8050b9ccbe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a64621b72c149d08a24e8050b9ccbe9" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 42169 } }
I20260928 06:40:53.318235 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 29441
I20260928 06:40:53.318279 29509 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a64621b72c149d08a24e8050b9ccbe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:53.318328 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:40:53.318331 29509 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a64621b72c149d08a24e8050b9ccbe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a64621b72c149d08a24e8050b9ccbe9, State: Initialized, Role: FOLLOWER
I20260928 06:40:53.318506 19277 external_mini_cluster.cc:1428] Setting key 4f4a3a6fb77a27fd5f3779aecf669752
I20260928 06:40:53.318477 29509 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a64621b72c149d08a24e8050b9ccbe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a64621b72c149d08a24e8050b9ccbe9" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 42169 } }
I20260928 06:40:53.318554 29509 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a64621b72c149d08a24e8050b9ccbe9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:40:53.318593 29509 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a64621b72c149d08a24e8050b9ccbe9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:40:53.318640 29509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a64621b72c149d08a24e8050b9ccbe9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:53.319319 29509 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a64621b72c149d08a24e8050b9ccbe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a64621b72c149d08a24e8050b9ccbe9" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 42169 } }
I20260928 06:40:53.319434 29509 leader_election.cc:302] T 00000000000000000000000000000000 P 4a64621b72c149d08a24e8050b9ccbe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a64621b72c149d08a24e8050b9ccbe9; no voters:
I20260928 06:40:53.319593 29509 leader_election.cc:288] T 00000000000000000000000000000000 P 4a64621b72c149d08a24e8050b9ccbe9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:40:53.319625 29514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a64621b72c149d08a24e8050b9ccbe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:53.319788 29514 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a64621b72c149d08a24e8050b9ccbe9 [term 1 LEADER]: Becoming Leader. State: Replica: 4a64621b72c149d08a24e8050b9ccbe9, State: Running, Role: LEADER
I20260928 06:40:53.319832 29509 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a64621b72c149d08a24e8050b9ccbe9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:40:53.319878 29514 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a64621b72c149d08a24e8050b9ccbe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a64621b72c149d08a24e8050b9ccbe9" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 42169 } }
I20260928 06:40:53.320817 29516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a64621b72c149d08a24e8050b9ccbe9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a64621b72c149d08a24e8050b9ccbe9. Latest consensus state: current_term: 1 leader_uuid: "4a64621b72c149d08a24e8050b9ccbe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a64621b72c149d08a24e8050b9ccbe9" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 42169 } } }
I20260928 06:40:53.320906 29516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a64621b72c149d08a24e8050b9ccbe9 [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:53.321091 29527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:40:53.321384 29515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a64621b72c149d08a24e8050b9ccbe9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a64621b72c149d08a24e8050b9ccbe9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a64621b72c149d08a24e8050b9ccbe9" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 42169 } } }
I20260928 06:40:53.321547 29515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a64621b72c149d08a24e8050b9ccbe9 [sys.catalog]: This master's current role is: LEADER
I20260928 06:40:53.321683 29527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:40:53.323325 29527 catalog_manager.cc:1347] Generated new cluster ID: 521d7a0495ec449b9572f4eeacec783c
I20260928 06:40:53.323383 29527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:40:53 dist-test-slave-b28r krb5kdc[29428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:53.328935 29513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:53.318966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46116 (local address 127.18.211.126:42169)
0928 06:40:53.319552 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:53.319562 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:53.319597 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:40:53.319719 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:53.319727 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:53.319801 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:53.319978 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:53.320766 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.322034 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.323167 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.323711 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.323818 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:53.325966 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:53.325971 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:53.325971 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:53.325983 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:53.327537 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:53.327867 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:53.327870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:53.327870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:53.327974 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:53.328151 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:53.328152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:53.328152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:53.328388 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:53.328540 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:53.328633 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 06:40:53.328790 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":451,"threads_started":1}
I20260928 06:40:53.338120 29527 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:40:53.338604 29527 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:40:53.346068 29527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a64621b72c149d08a24e8050b9ccbe9: Generated new TSK 0
I20260928 06:40:53.346235 29527 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.211.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab.
Entry for principal kudu/127.18.211.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab.
I20260928 06:40:53.373728 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.65:0
--local_ip_for_outbound_sockets=127.18.211.65
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:42169
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:53.455803 29535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:53.456013 29535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:53.456036 29535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:53.457156 29535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:53.457201 29535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:53.457226 29535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.65
I20260928 06:40:53.458700 29535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:42169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.65
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:53.458936 29535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:53.459189 29535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:53.464010 29540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:53.464025 29541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:53.464026 29543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:53.464303 29535 server_base.cc:1034] running on GCE node
I20260928 06:40:53.464505 29535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:53.464732 29535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:53.465943 29535 hybrid_clock.cc:648] HybridClock initialized: now 1790577653465929 us; error 31 us; skew 500 ppm
Sep 28 06:40:53 dist-test-slave-b28r krb5kdc[29428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577653, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:53.468621 29535 init.cc:377] Logged in from keytab as kudu/127.18.211.65@KRBTEST.COM (short username kudu)
I20260928 06:40:53.469071 29535 webserver.cc:460] Webserver started at http://127.18.211.65:44869/ using document root <none> and password file <none>
I20260928 06:40:53.469344 29535 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:53.469395 29535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:53.469516 29535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:53.470553 29535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/data/instance:
uuid: "79ef270434c64b7a96b29949ee465db6"
format_stamp: "Formatted at 2026-09-28 06:40:53 on dist-test-slave-b28r"
server_key: "c4565ba2dc7da1cef2d17e9640674071"
server_key_iv: "6151e1d7472d619bda565d3ccefa56bf"
server_key_version: "encryptionkey@0"
I20260928 06:40:53.470880 29535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance:
uuid: "79ef270434c64b7a96b29949ee465db6"
format_stamp: "Formatted at 2026-09-28 06:40:53 on dist-test-slave-b28r"
server_key: "c4565ba2dc7da1cef2d17e9640674071"
server_key_iv: "6151e1d7472d619bda565d3ccefa56bf"
server_key_version: "encryptionkey@0"
I20260928 06:40:53.472196 29535 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:40:53.472980 29550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:53.473194 29535 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:40:53.473286 29535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/wal
uuid: "79ef270434c64b7a96b29949ee465db6"
format_stamp: "Formatted at 2026-09-28 06:40:53 on dist-test-slave-b28r"
server_key: "c4565ba2dc7da1cef2d17e9640674071"
server_key_iv: "6151e1d7472d619bda565d3ccefa56bf"
server_key_version: "encryptionkey@0"
I20260928 06:40:53.473354 29535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:53.490159 29535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:53.492545 29535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:53.492703 29535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:53.492944 29535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:53.493284 29535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:53.493325 29535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:53.493359 29535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:53.493396 29535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:53.499473 29535 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.65:46369
I20260928 06:40:53.499516 29663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.65:46369 every 8 connection(s)
I20260928 06:40:53.499851 29535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:40:53 dist-test-slave-b28r krb5kdc[29428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577653, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:53.508617 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 29535
I20260928 06:40:53.508728 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance
I20260928 06:40:53.508949 19277 external_mini_cluster.cc:1428] Setting key ee7c7188f6578be4d8fb54bc6a4d6a5b
I20260928 06:40:53.509150 29513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:53.500604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:49885 (local address 127.18.211.126:42169)
0928 06:40:53.500670 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:53.500672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:53.500864 (+ 192us) server_negotiation.cc:407] Connection header received
0928 06:40:53.501374 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:53.501376 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:53.501391 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:53.501459 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:53.502415 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.502925 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.503978 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.504228 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.504292 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:53.506355 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:53.506356 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:53.506357 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:53.506365 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:53.507837 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:53.508217 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:53.508218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:53.508218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:53.508286 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:53.508489 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:53.508490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:53.508491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:53.508628 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:53.508703 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:53.508971 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 06:40:53.509066 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:40:53.509389 29666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:53.500566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:42169 (local address 127.18.211.65:49885)
0928 06:40:53.500739 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:53.500783 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:53.501264 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:53.501525 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:53.501528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:53.502085 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:53.502317 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:53.502323 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.503007 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.503008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:53.503893 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:53.503896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.504325 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.504326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:53.504434 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:53.504885 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:53.504893 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:53.506243 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:53.507915 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:53.507921 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:53.507923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:53.508131 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:53.508345 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:53.508346 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:53.508346 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:53.508433 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:53.508688 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:53.508693 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:53.508844 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 06:40:53.508929 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 06:40:53.509073 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":88,"threads_started":1}
I20260928 06:40:53.509848 29664 heartbeater.cc:347] Connected to a master server at 127.18.211.126:42169
I20260928 06:40:53.509945 29664 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:53.510113 29664 heartbeater.cc:511] Master 127.18.211.126:42169 requested a full tablet report, sending...
I20260928 06:40:53.510615 29473 ts_manager.cc:194] Registered new tserver with Master: 79ef270434c64b7a96b29949ee465db6 (127.18.211.65:46369)
I20260928 06:40:53.511264 29473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.65@KRBTEST.COM'} at 127.18.211.65:49885
WARNING: no policy specified for kudu/127.18.211.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab.
Entry for principal kudu/127.18.211.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab.
I20260928 06:40:53.531692 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.66:0
--local_ip_for_outbound_sockets=127.18.211.66
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:42169
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:53.610677 29669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:53.610877 29669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:53.610898 29669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:53.612048 29669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:53.612089 29669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:53.612116 29669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.66
I20260928 06:40:53.613559 29669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:42169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.66
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:53.613771 29669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:53.613981 29669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:53.618793 29674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:53.618799 29675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:53.618829 29677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:53.618958 29669 server_base.cc:1034] running on GCE node
I20260928 06:40:53.619218 29669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:53.619459 29669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:53.620554 29669 hybrid_clock.cc:648] HybridClock initialized: now 1790577653620541 us; error 30 us; skew 500 ppm
Sep 28 06:40:53 dist-test-slave-b28r krb5kdc[29428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577653, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:53.623127 29669 init.cc:377] Logged in from keytab as kudu/127.18.211.66@KRBTEST.COM (short username kudu)
I20260928 06:40:53.623605 29669 webserver.cc:460] Webserver started at http://127.18.211.66:46475/ using document root <none> and password file <none>
I20260928 06:40:53.623818 29669 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:53.623867 29669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:53.623989 29669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:53.625028 29669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/data/instance:
uuid: "a998ff5c068c49518d6874ea7b872ac6"
format_stamp: "Formatted at 2026-09-28 06:40:53 on dist-test-slave-b28r"
server_key: "27714cc5b2f4d076ed52435dd8b06f00"
server_key_iv: "81b818df807b7010cb11d4cd16a8d6cb"
server_key_version: "encryptionkey@0"
I20260928 06:40:53.625363 29669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance:
uuid: "a998ff5c068c49518d6874ea7b872ac6"
format_stamp: "Formatted at 2026-09-28 06:40:53 on dist-test-slave-b28r"
server_key: "27714cc5b2f4d076ed52435dd8b06f00"
server_key_iv: "81b818df807b7010cb11d4cd16a8d6cb"
server_key_version: "encryptionkey@0"
I20260928 06:40:53.626605 29669 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:40:53.627372 29684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:53.627512 29669 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:40:53.627595 29669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/wal
uuid: "a998ff5c068c49518d6874ea7b872ac6"
format_stamp: "Formatted at 2026-09-28 06:40:53 on dist-test-slave-b28r"
server_key: "27714cc5b2f4d076ed52435dd8b06f00"
server_key_iv: "81b818df807b7010cb11d4cd16a8d6cb"
server_key_version: "encryptionkey@0"
I20260928 06:40:53.627660 29669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:53.633809 29669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:53.635492 29669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:53.635618 29669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:53.635819 29669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:53.636082 29669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:53.636111 29669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:53.636130 29669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:53.636142 29669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:53.641523 29669 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.66:44841
I20260928 06:40:53.641578 29797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.66:44841 every 8 connection(s)
I20260928 06:40:53.641855 29669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
I20260928 06:40:53.645951 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 29669
I20260928 06:40:53.646049 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance
I20260928 06:40:53.646225 19277 external_mini_cluster.cc:1428] Setting key 0d5b66ef98defa5cc7786977f29a452a
Sep 28 06:40:53 dist-test-slave-b28r krb5kdc[29428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577653, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:53.650516 29513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:53.642487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:52171 (local address 127.18.211.126:42169)
0928 06:40:53.642566 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:53.642568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:53.642700 (+ 132us) server_negotiation.cc:407] Connection header received
0928 06:40:53.643191 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:53.643192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:53.643203 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:53.643252 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:53.644120 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.644622 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.645594 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.645791 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.645848 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:53.648038 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:53.648040 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:53.648040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:53.648048 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:53.649299 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:53.649660 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:53.649661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:53.649662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:53.649714 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:53.649932 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:53.649932 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:53.649933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:53.650036 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:53.650091 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:53.650347 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 06:40:53.650427 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 06:40:53.650689 29800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:53.642453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:42169 (local address 127.18.211.66:52171)
0928 06:40:53.642595 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:53.642628 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:53.643103 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:53.643305 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:53.643308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:53.643854 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:53.644041 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:53.644045 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.644698 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.644699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:53.645531 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:53.645534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.645860 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.645861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:53.645949 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:53.646443 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:53.646453 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:53.647811 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:53.649375 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:53.649381 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:53.649384 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:53.649595 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:53.649775 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:53.649776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:53.649776 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:53.649877 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:53.650090 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:53.650097 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:53.650230 (+ 133us) client_negotiation.cc:769] Sending connection context
0928 06:40:53.650319 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 06:40:53.650471 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":68,"threads_started":1}
I20260928 06:40:53.651146 29798 heartbeater.cc:347] Connected to a master server at 127.18.211.126:42169
I20260928 06:40:53.651248 29798 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:53.651440 29798 heartbeater.cc:511] Master 127.18.211.126:42169 requested a full tablet report, sending...
I20260928 06:40:53.651798 29473 ts_manager.cc:194] Registered new tserver with Master: a998ff5c068c49518d6874ea7b872ac6 (127.18.211.66:44841)
I20260928 06:40:53.652148 29473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.66@KRBTEST.COM'} at 127.18.211.66:52171
WARNING: no policy specified for kudu/127.18.211.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab.
Entry for principal kudu/127.18.211.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab.
I20260928 06:40:53.669003 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.67:0
--local_ip_for_outbound_sockets=127.18.211.67
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:42169
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:40:53.750795 29803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:40:53.750994 29803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:40:53.751016 29803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:40:53.752163 29803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:40:53.752204 29803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:40:53.752229 29803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.67
I20260928 06:40:53.753652 29803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:42169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.67
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:40:53.753831 29803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:40:53.754005 29803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:40:53.758665 29809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:53.758659 29811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:40:53.758658 29808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:40:53.758952 29803 server_base.cc:1034] running on GCE node
I20260928 06:40:53.759164 29803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:40:53.759369 29803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:40:53.760497 29803 hybrid_clock.cc:648] HybridClock initialized: now 1790577653760481 us; error 33 us; skew 500 ppm
Sep 28 06:40:53 dist-test-slave-b28r krb5kdc[29428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577653, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:40:53.762748 29803 init.cc:377] Logged in from keytab as kudu/127.18.211.67@KRBTEST.COM (short username kudu)
I20260928 06:40:53.763217 29803 webserver.cc:460] Webserver started at http://127.18.211.67:41613/ using document root <none> and password file <none>
I20260928 06:40:53.763449 29803 fs_manager.cc:359] Metadata directory not provided
I20260928 06:40:53.763499 29803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:40:53.763624 29803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:40:53.764662 29803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/data/instance:
uuid: "05af95f99f7f4f789524bb34975ec19f"
format_stamp: "Formatted at 2026-09-28 06:40:53 on dist-test-slave-b28r"
server_key: "b4503437e0384b2f880f83e29cc54b9e"
server_key_iv: "31268ef35a03b35c5817dbd526476524"
server_key_version: "encryptionkey@0"
I20260928 06:40:53.764986 29803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance:
uuid: "05af95f99f7f4f789524bb34975ec19f"
format_stamp: "Formatted at 2026-09-28 06:40:53 on dist-test-slave-b28r"
server_key: "b4503437e0384b2f880f83e29cc54b9e"
server_key_iv: "31268ef35a03b35c5817dbd526476524"
server_key_version: "encryptionkey@0"
I20260928 06:40:53.766218 29803 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:40:53.766920 29818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:53.767074 29803 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:40:53.767181 29803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/wal
uuid: "05af95f99f7f4f789524bb34975ec19f"
format_stamp: "Formatted at 2026-09-28 06:40:53 on dist-test-slave-b28r"
server_key: "b4503437e0384b2f880f83e29cc54b9e"
server_key_iv: "31268ef35a03b35c5817dbd526476524"
server_key_version: "encryptionkey@0"
I20260928 06:40:53.767241 29803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:40:53.779403 29803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:40:53.781209 29803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:40:53.781345 29803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:40:53.781548 29803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:40:53.781831 29803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:40:53.781863 29803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:53.781889 29803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:40:53.781908 29803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:40:53.787309 29803 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.67:33869
I20260928 06:40:53.787369 29931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.67:33869 every 8 connection(s)
I20260928 06:40:53.787652 29803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:40:53 dist-test-slave-b28r krb5kdc[29428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577653, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:40:53.793499 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 29803
I20260928 06:40:53.793612 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance
I20260928 06:40:53.793813 19277 external_mini_cluster.cc:1428] Setting key 9e7a1e1dca126105a225a9c8b6ef61b4
I20260928 06:40:53.796407 29513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:53.788355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:34757 (local address 127.18.211.126:42169)
0928 06:40:53.788412 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:53.788414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:53.788638 (+ 224us) server_negotiation.cc:407] Connection header received
0928 06:40:53.789129 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:53.789130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:53.789140 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:53.789178 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:53.790099 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.790631 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.791698 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.791896 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.791957 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:53.794023 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:53.794025 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:53.794025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:53.794033 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:53.795370 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:53.795696 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:53.795696 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:53.795697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:53.795742 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:53.795935 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:53.795936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:53.795936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:53.796026 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:53.796069 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:53.796255 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 06:40:53.796345 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:40:53.796528 29934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:53.788320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:42169 (local address 127.18.211.67:34757)
0928 06:40:53.788517 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:53.788555 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:53.789040 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:53.789233 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:53.789236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:53.789773 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:40:53.789990 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:53.789995 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.790729 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.790730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:53.791624 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:53.791626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.791970 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.791971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:53.792071 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:53.792500 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:40:53.792507 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:40:53.793916 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:40:53.795431 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:53.795437 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:53.795439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:53.795642 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:53.795803 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:40:53.795803 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:40:53.795803 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:40:53.795894 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:40:53.796065 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:40:53.796068 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:40:53.796169 (+ 101us) client_negotiation.cc:769] Sending connection context
0928 06:40:53.796242 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 06:40:53.796356 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":93,"threads_started":1}
I20260928 06:40:53.796991 29932 heartbeater.cc:347] Connected to a master server at 127.18.211.126:42169
I20260928 06:40:53.797076 29932 heartbeater.cc:464] Registering TS with master...
I20260928 06:40:53.797240 29932 heartbeater.cc:511] Master 127.18.211.126:42169 requested a full tablet report, sending...
I20260928 06:40:53.797608 29473 ts_manager.cc:194] Registered new tserver with Master: 05af95f99f7f4f789524bb34975ec19f (127.18.211.67:33869)
I20260928 06:40:53.797938 29473 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.67@KRBTEST.COM'} at 127.18.211.67:34757
I20260928 06:40:53.805258 19277 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:40:53.812218 29513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:53.806557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46120 (local address 127.18.211.126:42169)
0928 06:40:53.806630 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:53.806632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:53.806694 (+ 62us) server_negotiation.cc:407] Connection header received
0928 06:40:53.806771 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:53.806772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:53.806782 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:53.806824 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:40:53.807411 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.807892 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.808628 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.808821 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.808877 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:53.809922 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:40:53.809923 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:40:53.809923 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:40:53.809929 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:40:53.811019 (+ 1090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:53.811449 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:53.811450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:53.811450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:53.811501 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:40:53.811711 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:40:53.811712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:40:53.811713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:40:53.811864 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:40:53.811947 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:53.812094 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 06:40:53.812157 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 06:40:53.813633 29473 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:40:53.815281 29473 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:40:53.821310 29941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:53.817287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:33869 (local address 127.0.0.1:58930)
0928 06:40:53.817608 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:53.817617 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:53.817657 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:53.818048 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:53.818049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:53.818051 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:53.818083 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:53.818084 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.818987 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.818988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:53.820222 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:53.820225 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.821040 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.821041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:53.821114 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:53.821129 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 06:40:53.821187 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 06:40:53.821236 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":70,"threads_started":1}
I20260928 06:40:53.821326 29942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:53.817423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58930 (local address 127.18.211.67:33869)
0928 06:40:53.817843 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:53.817848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:53.817857 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:40:53.817888 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:53.817890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:53.817907 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:53.817984 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:53.818150 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.818897 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.820324 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.820949 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.821021 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:53.821173 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:53.821201 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 06:40:53.821241 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":86,"threads_started":1}
I20260928 06:40:53.821491 29940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:53.817284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:44841 (local address 127.0.0.1:36406)
0928 06:40:53.817462 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:53.817509 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:53.817575 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:53.817924 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:53.817928 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:53.817931 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:53.817998 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:53.818000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.819388 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.819389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:53.820437 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:53.820439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.821296 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.821297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:53.821353 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:53.821356 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:40:53.821384 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 06:40:53.821422 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":56,"threads_started":1}
I20260928 06:40:53.821627 29944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:53.817480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:46369 (local address 127.0.0.1:40562)
0928 06:40:53.817658 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:53.817668 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:53.817720 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:53.817990 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:53.817991 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:53.817993 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:53.818045 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:53.818109 (+ 64us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.819481 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.819482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:53.820339 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:53.820341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.821451 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.821452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:53.821504 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:53.821507 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:40:53.821537 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 06:40:53.821569 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":50,"threads_started":1}
I20260928 06:40:53.821951 29945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:53.817649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40562 (local address 127.18.211.65:46369)
0928 06:40:53.817828 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:53.817831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:53.817841 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:40:53.817881 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:53.817883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:53.817897 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:53.818184 (+ 287us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:53.818262 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.819392 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.820409 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.821197 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.821744 (+ 547us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:53.821814 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:53.821839 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 06:40:53.821876 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":88,"threads_started":1}
I20260928 06:40:53.821956 29866 tablet_service.cc:1467] Processing CreateTablet for tablet 2ac4718f25644ef0be087a16fa69f71e (DEFAULT_TABLE table=test-table [id=515da95eec2b4d0db7701775fb4537d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:53.822268 29943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:53.817448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36406 (local address 127.18.211.66:44841)
0928 06:40:53.817759 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:53.817765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:53.817775 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:40:53.817814 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:53.817816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:53.817831 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:53.817911 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:53.818064 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.819277 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.820521 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.821197 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.822059 (+ 862us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:53.822123 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:53.822145 (+ 22us) server_negotiation.cc:299] Negotiation successful
0928 06:40:53.822194 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":83,"threads_started":1}
I20260928 06:40:53.822297 29866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ac4718f25644ef0be087a16fa69f71e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:53.822574 29598 tablet_service.cc:1467] Processing CreateTablet for tablet 2ac4718f25644ef0be087a16fa69f71e (DEFAULT_TABLE table=test-table [id=515da95eec2b4d0db7701775fb4537d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:53.822805 29732 tablet_service.cc:1467] Processing CreateTablet for tablet 2ac4718f25644ef0be087a16fa69f71e (DEFAULT_TABLE table=test-table [id=515da95eec2b4d0db7701775fb4537d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:40:53.822921 29598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ac4718f25644ef0be087a16fa69f71e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:53.823122 29732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ac4718f25644ef0be087a16fa69f71e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:40:53.824972 29946 tablet_bootstrap.cc:492] T 2ac4718f25644ef0be087a16fa69f71e P 05af95f99f7f4f789524bb34975ec19f: Bootstrap starting.
I20260928 06:40:53.825637 29946 tablet_bootstrap.cc:654] T 2ac4718f25644ef0be087a16fa69f71e P 05af95f99f7f4f789524bb34975ec19f: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:53.826067 29947 tablet_bootstrap.cc:492] T 2ac4718f25644ef0be087a16fa69f71e P 79ef270434c64b7a96b29949ee465db6: Bootstrap starting.
I20260928 06:40:53.826133 29946 log.cc:824] T 2ac4718f25644ef0be087a16fa69f71e P 05af95f99f7f4f789524bb34975ec19f: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:53.826261 29948 tablet_bootstrap.cc:492] T 2ac4718f25644ef0be087a16fa69f71e P a998ff5c068c49518d6874ea7b872ac6: Bootstrap starting.
I20260928 06:40:53.826787 29947 tablet_bootstrap.cc:654] T 2ac4718f25644ef0be087a16fa69f71e P 79ef270434c64b7a96b29949ee465db6: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:53.826864 29948 tablet_bootstrap.cc:654] T 2ac4718f25644ef0be087a16fa69f71e P a998ff5c068c49518d6874ea7b872ac6: Neither blocks nor log segments found. Creating new log.
I20260928 06:40:53.827049 29946 tablet_bootstrap.cc:492] T 2ac4718f25644ef0be087a16fa69f71e P 05af95f99f7f4f789524bb34975ec19f: No bootstrap required, opened a new log
I20260928 06:40:53.827147 29946 ts_tablet_manager.cc:1397] T 2ac4718f25644ef0be087a16fa69f71e P 05af95f99f7f4f789524bb34975ec19f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:40:53.827378 29948 log.cc:824] T 2ac4718f25644ef0be087a16fa69f71e P a998ff5c068c49518d6874ea7b872ac6: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:53.827409 29947 log.cc:824] T 2ac4718f25644ef0be087a16fa69f71e P 79ef270434c64b7a96b29949ee465db6: Log is configured to *not* fsync() on all Append() calls
I20260928 06:40:53.828127 29948 tablet_bootstrap.cc:492] T 2ac4718f25644ef0be087a16fa69f71e P a998ff5c068c49518d6874ea7b872ac6: No bootstrap required, opened a new log
I20260928 06:40:53.828197 29948 ts_tablet_manager.cc:1397] T 2ac4718f25644ef0be087a16fa69f71e P a998ff5c068c49518d6874ea7b872ac6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:40:53.829020 29946 raft_consensus.cc:348] T 2ac4718f25644ef0be087a16fa69f71e P 05af95f99f7f4f789524bb34975ec19f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a998ff5c068c49518d6874ea7b872ac6" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 44841 } } peers { permanent_uuid: "05af95f99f7f4f789524bb34975ec19f" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 33869 } } peers { permanent_uuid: "79ef270434c64b7a96b29949ee465db6" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 46369 } }
I20260928 06:40:53.829186 29946 raft_consensus.cc:374] T 2ac4718f25644ef0be087a16fa69f71e P 05af95f99f7f4f789524bb34975ec19f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:53.829221 29946 raft_consensus.cc:729] T 2ac4718f25644ef0be087a16fa69f71e P 05af95f99f7f4f789524bb34975ec19f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05af95f99f7f4f789524bb34975ec19f, State: Initialized, Role: FOLLOWER
I20260928 06:40:53.829357 29946 consensus_queue.cc:260] T 2ac4718f25644ef0be087a16fa69f71e P 05af95f99f7f4f789524bb34975ec19f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a998ff5c068c49518d6874ea7b872ac6" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 44841 } } peers { permanent_uuid: "05af95f99f7f4f789524bb34975ec19f" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 33869 } } peers { permanent_uuid: "79ef270434c64b7a96b29949ee465db6" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 46369 } }
I20260928 06:40:53.829634 29946 ts_tablet_manager.cc:1428] T 2ac4718f25644ef0be087a16fa69f71e P 05af95f99f7f4f789524bb34975ec19f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:40:53.829691 29932 heartbeater.cc:503] Master 127.18.211.126:42169 was elected leader, sending a full tablet report...
I20260928 06:40:53.830046 29948 raft_consensus.cc:348] T 2ac4718f25644ef0be087a16fa69f71e P a998ff5c068c49518d6874ea7b872ac6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a998ff5c068c49518d6874ea7b872ac6" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 44841 } } peers { permanent_uuid: "05af95f99f7f4f789524bb34975ec19f" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 33869 } } peers { permanent_uuid: "79ef270434c64b7a96b29949ee465db6" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 46369 } }
I20260928 06:40:53.830201 29948 raft_consensus.cc:374] T 2ac4718f25644ef0be087a16fa69f71e P a998ff5c068c49518d6874ea7b872ac6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:53.830240 29948 raft_consensus.cc:729] T 2ac4718f25644ef0be087a16fa69f71e P a998ff5c068c49518d6874ea7b872ac6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a998ff5c068c49518d6874ea7b872ac6, State: Initialized, Role: FOLLOWER
I20260928 06:40:53.830351 29948 consensus_queue.cc:260] T 2ac4718f25644ef0be087a16fa69f71e P a998ff5c068c49518d6874ea7b872ac6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a998ff5c068c49518d6874ea7b872ac6" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 44841 } } peers { permanent_uuid: "05af95f99f7f4f789524bb34975ec19f" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 33869 } } peers { permanent_uuid: "79ef270434c64b7a96b29949ee465db6" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 46369 } }
I20260928 06:40:53.830613 29948 ts_tablet_manager.cc:1428] T 2ac4718f25644ef0be087a16fa69f71e P a998ff5c068c49518d6874ea7b872ac6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:40:53.830687 29798 heartbeater.cc:503] Master 127.18.211.126:42169 was elected leader, sending a full tablet report...
I20260928 06:40:53.830981 29947 tablet_bootstrap.cc:492] T 2ac4718f25644ef0be087a16fa69f71e P 79ef270434c64b7a96b29949ee465db6: No bootstrap required, opened a new log
I20260928 06:40:53.831049 29947 ts_tablet_manager.cc:1397] T 2ac4718f25644ef0be087a16fa69f71e P 79ef270434c64b7a96b29949ee465db6: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260928 06:40:53.832513 29947 raft_consensus.cc:348] T 2ac4718f25644ef0be087a16fa69f71e P 79ef270434c64b7a96b29949ee465db6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a998ff5c068c49518d6874ea7b872ac6" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 44841 } } peers { permanent_uuid: "05af95f99f7f4f789524bb34975ec19f" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 33869 } } peers { permanent_uuid: "79ef270434c64b7a96b29949ee465db6" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 46369 } }
I20260928 06:40:53.832631 29947 raft_consensus.cc:374] T 2ac4718f25644ef0be087a16fa69f71e P 79ef270434c64b7a96b29949ee465db6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:40:53.832664 29947 raft_consensus.cc:729] T 2ac4718f25644ef0be087a16fa69f71e P 79ef270434c64b7a96b29949ee465db6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79ef270434c64b7a96b29949ee465db6, State: Initialized, Role: FOLLOWER
I20260928 06:40:53.832767 29947 consensus_queue.cc:260] T 2ac4718f25644ef0be087a16fa69f71e P 79ef270434c64b7a96b29949ee465db6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a998ff5c068c49518d6874ea7b872ac6" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 44841 } } peers { permanent_uuid: "05af95f99f7f4f789524bb34975ec19f" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 33869 } } peers { permanent_uuid: "79ef270434c64b7a96b29949ee465db6" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 46369 } }
I20260928 06:40:53.833029 29947 ts_tablet_manager.cc:1428] T 2ac4718f25644ef0be087a16fa69f71e P 79ef270434c64b7a96b29949ee465db6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:40:53.833060 29664 heartbeater.cc:503] Master 127.18.211.126:42169 was elected leader, sending a full tablet report...
I20260928 06:40:53.882793 29953 raft_consensus.cc:482] T 2ac4718f25644ef0be087a16fa69f71e P a998ff5c068c49518d6874ea7b872ac6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:40:53.882907 29953 raft_consensus.cc:504] T 2ac4718f25644ef0be087a16fa69f71e P a998ff5c068c49518d6874ea7b872ac6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a998ff5c068c49518d6874ea7b872ac6" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 44841 } } peers { permanent_uuid: "05af95f99f7f4f789524bb34975ec19f" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 33869 } } peers { permanent_uuid: "79ef270434c64b7a96b29949ee465db6" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 46369 } }
I20260928 06:40:53.883255 29953 leader_election.cc:288] T 2ac4718f25644ef0be087a16fa69f71e P a998ff5c068c49518d6874ea7b872ac6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05af95f99f7f4f789524bb34975ec19f (127.18.211.67:33869), 79ef270434c64b7a96b29949ee465db6 (127.18.211.65:46369)
I20260928 06:40:53.885994 29800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:53.883409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:46369 (local address 127.18.211.66:47649)
0928 06:40:53.883471 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:53.883482 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:53.883522 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:53.883716 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:53.883717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:53.883727 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:53.883802 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:53.883804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.884468 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.884469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:53.885205 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:53.885207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.885735 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.885736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:53.885814 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:53.885817 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:40:53.885862 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 06:40:53.885912 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 06:40:53.885984 29945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:53.883558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:47649 (local address 127.18.211.65:46369)
0928 06:40:53.883599 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:53.883601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:53.883609 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:40:53.883635 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:53.883635 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:53.883642 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:53.883699 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:53.883865 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.884389 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.885291 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.885659 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.885719 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:53.885728 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:53.885877 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 06:40:53.885914 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:40:53.885998 29942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:53.883484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:57033 (local address 127.18.211.67:33869)
0928 06:40:53.883534 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:53.883536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:53.883691 (+ 155us) server_negotiation.cc:407] Connection header received
0928 06:40:53.883722 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:53.883723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:53.883732 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:53.883780 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:40:53.883918 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.884432 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.885316 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.885697 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.885756 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:53.885764 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:53.885897 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 06:40:53.885929 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 06:40:53.886015 29955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:53.883409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:33869 (local address 127.18.211.66:57033)
0928 06:40:53.883624 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:53.883631 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 06:40:53.883669 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:40:53.883788 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:40:53.883789 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:40:53.883790 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:40:53.883858 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:53.883859 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.884506 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.884507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:53.885237 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:40:53.885239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.885762 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.885763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:40:53.885833 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:53.885837 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:40:53.885891 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 06:40:53.885929 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":14,"thread_start_us":97,"threads_started":1}
I20260928 06:40:53.886332 29618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac4718f25644ef0be087a16fa69f71e" candidate_uuid: "a998ff5c068c49518d6874ea7b872ac6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79ef270434c64b7a96b29949ee465db6" is_pre_election: true
I20260928 06:40:53.886502 29618 raft_consensus.cc:2454] T 2ac4718f25644ef0be087a16fa69f71e P 79ef270434c64b7a96b29949ee465db6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a998ff5c068c49518d6874ea7b872ac6 in term 0.
I20260928 06:40:53.886483 29886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac4718f25644ef0be087a16fa69f71e" candidate_uuid: "a998ff5c068c49518d6874ea7b872ac6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05af95f99f7f4f789524bb34975ec19f" is_pre_election: true
I20260928 06:40:53.886610 29886 raft_consensus.cc:2454] T 2ac4718f25644ef0be087a16fa69f71e P 05af95f99f7f4f789524bb34975ec19f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a998ff5c068c49518d6874ea7b872ac6 in term 0.
I20260928 06:40:53.886735 29688 leader_election.cc:302] T 2ac4718f25644ef0be087a16fa69f71e P a998ff5c068c49518d6874ea7b872ac6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79ef270434c64b7a96b29949ee465db6, a998ff5c068c49518d6874ea7b872ac6; no voters:
I20260928 06:40:53.886857 29953 raft_consensus.cc:2781] T 2ac4718f25644ef0be087a16fa69f71e P a998ff5c068c49518d6874ea7b872ac6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:40:53.886919 29953 raft_consensus.cc:482] T 2ac4718f25644ef0be087a16fa69f71e P a998ff5c068c49518d6874ea7b872ac6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:40:53.886935 29953 raft_consensus.cc:3037] T 2ac4718f25644ef0be087a16fa69f71e P a998ff5c068c49518d6874ea7b872ac6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:53.887588 29953 raft_consensus.cc:504] T 2ac4718f25644ef0be087a16fa69f71e P a998ff5c068c49518d6874ea7b872ac6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a998ff5c068c49518d6874ea7b872ac6" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 44841 } } peers { permanent_uuid: "05af95f99f7f4f789524bb34975ec19f" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 33869 } } peers { permanent_uuid: "79ef270434c64b7a96b29949ee465db6" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 46369 } }
I20260928 06:40:53.887704 29953 leader_election.cc:288] T 2ac4718f25644ef0be087a16fa69f71e P a998ff5c068c49518d6874ea7b872ac6 [CANDIDATE]: Term 1 election: Requested vote from peers 05af95f99f7f4f789524bb34975ec19f (127.18.211.67:33869), 79ef270434c64b7a96b29949ee465db6 (127.18.211.65:46369)
I20260928 06:40:53.887842 29886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac4718f25644ef0be087a16fa69f71e" candidate_uuid: "a998ff5c068c49518d6874ea7b872ac6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05af95f99f7f4f789524bb34975ec19f"
I20260928 06:40:53.887887 29618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac4718f25644ef0be087a16fa69f71e" candidate_uuid: "a998ff5c068c49518d6874ea7b872ac6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79ef270434c64b7a96b29949ee465db6"
I20260928 06:40:53.887923 29886 raft_consensus.cc:3037] T 2ac4718f25644ef0be087a16fa69f71e P 05af95f99f7f4f789524bb34975ec19f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:53.887957 29618 raft_consensus.cc:3037] T 2ac4718f25644ef0be087a16fa69f71e P 79ef270434c64b7a96b29949ee465db6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:40:53.888593 29886 raft_consensus.cc:2454] T 2ac4718f25644ef0be087a16fa69f71e P 05af95f99f7f4f789524bb34975ec19f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a998ff5c068c49518d6874ea7b872ac6 in term 1.
I20260928 06:40:53.888676 29618 raft_consensus.cc:2454] T 2ac4718f25644ef0be087a16fa69f71e P 79ef270434c64b7a96b29949ee465db6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a998ff5c068c49518d6874ea7b872ac6 in term 1.
I20260928 06:40:53.888767 29687 leader_election.cc:302] T 2ac4718f25644ef0be087a16fa69f71e P a998ff5c068c49518d6874ea7b872ac6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05af95f99f7f4f789524bb34975ec19f, a998ff5c068c49518d6874ea7b872ac6; no voters:
I20260928 06:40:53.888835 29953 raft_consensus.cc:2781] T 2ac4718f25644ef0be087a16fa69f71e P a998ff5c068c49518d6874ea7b872ac6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:40:53.888980 29953 raft_consensus.cc:686] T 2ac4718f25644ef0be087a16fa69f71e P a998ff5c068c49518d6874ea7b872ac6 [term 1 LEADER]: Becoming Leader. State: Replica: a998ff5c068c49518d6874ea7b872ac6, State: Running, Role: LEADER
I20260928 06:40:53.889060 29953 consensus_queue.cc:237] T 2ac4718f25644ef0be087a16fa69f71e P a998ff5c068c49518d6874ea7b872ac6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a998ff5c068c49518d6874ea7b872ac6" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 44841 } } peers { permanent_uuid: "05af95f99f7f4f789524bb34975ec19f" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 33869 } } peers { permanent_uuid: "79ef270434c64b7a96b29949ee465db6" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 46369 } }
I20260928 06:40:53.889896 29473 catalog_manager.cc:5495] T 2ac4718f25644ef0be087a16fa69f71e P a998ff5c068c49518d6874ea7b872ac6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a998ff5c068c49518d6874ea7b872ac6 (127.18.211.66). New cstate: current_term: 1 leader_uuid: "a998ff5c068c49518d6874ea7b872ac6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a998ff5c068c49518d6874ea7b872ac6" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 44841 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05af95f99f7f4f789524bb34975ec19f" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 33869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79ef270434c64b7a96b29949ee465db6" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 46369 } health_report { overall_health: UNKNOWN } } }
W20260928 06:40:53.892326 29799 tablet.cc:2369] T 2ac4718f25644ef0be087a16fa69f71e P a998ff5c068c49518d6874ea7b872ac6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:40:53.903035 29943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:53.900883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36414 (local address 127.18.211.66:44841)
0928 06:40:53.900922 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:40:53.900923 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:40:53.900932 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:40:53.900989 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:40:53.900989 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:40:53.901007 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:40:53.901080 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:40:53.901167 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.901646 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.902341 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:40:53.902512 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:40:53.902554 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:40:53.902750 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:40:53.902863 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:40:53.902920 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:40:53.902952 (+ 32us) server_negotiation.cc:299] Negotiation successful
0928 06:40:53.902967 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:40:53.904688 29886 raft_consensus.cc:1261] T 2ac4718f25644ef0be087a16fa69f71e P 05af95f99f7f4f789524bb34975ec19f [term 1 FOLLOWER]: Refusing update from remote peer a998ff5c068c49518d6874ea7b872ac6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:40:53.904721 29618 raft_consensus.cc:1261] T 2ac4718f25644ef0be087a16fa69f71e P 79ef270434c64b7a96b29949ee465db6 [term 1 FOLLOWER]: Refusing update from remote peer a998ff5c068c49518d6874ea7b872ac6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:40:53.905025 29953 consensus_queue.cc:1035] T 2ac4718f25644ef0be087a16fa69f71e P a998ff5c068c49518d6874ea7b872ac6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79ef270434c64b7a96b29949ee465db6" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 46369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:40:53.905153 29956 consensus_queue.cc:1035] T 2ac4718f25644ef0be087a16fa69f71e P a998ff5c068c49518d6874ea7b872ac6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05af95f99f7f4f789524bb34975ec19f" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 33869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:40:53.909480 29473 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:40:53.909560 29473 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:40:53.910218 29473 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a64621b72c149d08a24e8050b9ccbe9: Sending DeleteTablet for 3 replicas of tablet 2ac4718f25644ef0be087a16fa69f71e
I20260928 06:40:53.910557 29866 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ac4718f25644ef0be087a16fa69f71e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:53 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:58930
I20260928 06:40:53.910558 29598 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ac4718f25644ef0be087a16fa69f71e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:53 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:40562
I20260928 06:40:53.910789 29969 tablet_replica.cc:330] T 2ac4718f25644ef0be087a16fa69f71e P 79ef270434c64b7a96b29949ee465db6: stopping tablet replica
I20260928 06:40:53.910879 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 29535
I20260928 06:40:53.910907 29732 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ac4718f25644ef0be087a16fa69f71e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:40:53 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:36406
I20260928 06:40:53.911666 29970 tablet_replica.cc:330] T 2ac4718f25644ef0be087a16fa69f71e P a998ff5c068c49518d6874ea7b872ac6: stopping tablet replica
I20260928 06:40:53.911758 29970 raft_consensus.cc:2229] T 2ac4718f25644ef0be087a16fa69f71e P a998ff5c068c49518d6874ea7b872ac6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:40:53.911844 29970 raft_consensus.cc:2258] T 2ac4718f25644ef0be087a16fa69f71e P a998ff5c068c49518d6874ea7b872ac6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:53.913496 29970 ts_tablet_manager.cc:1905] T 2ac4718f25644ef0be087a16fa69f71e P a998ff5c068c49518d6874ea7b872ac6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:53.914986 29970 ts_tablet_manager.cc:1918] T 2ac4718f25644ef0be087a16fa69f71e P a998ff5c068c49518d6874ea7b872ac6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:40:53.915059 29970 log.cc:1196] T 2ac4718f25644ef0be087a16fa69f71e P a998ff5c068c49518d6874ea7b872ac6: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-1/wal/wals/2ac4718f25644ef0be087a16fa69f71e
I20260928 06:40:53.915318 29970 ts_tablet_manager.cc:1939] T 2ac4718f25644ef0be087a16fa69f71e P a998ff5c068c49518d6874ea7b872ac6: Deleting consensus metadata
I20260928 06:40:53.915668 29460 catalog_manager.cc:4841] TS a998ff5c068c49518d6874ea7b872ac6 (127.18.211.66:44841): tablet 2ac4718f25644ef0be087a16fa69f71e (table test-table [id=515da95eec2b4d0db7701775fb4537d5]) successfully deleted
I20260928 06:40:53.918286 29968 tablet_replica.cc:330] T 2ac4718f25644ef0be087a16fa69f71e P 05af95f99f7f4f789524bb34975ec19f: stopping tablet replica
W20260928 06:40:53.918298 29460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.211.65:46369 (error 108)
I20260928 06:40:53.918430 29968 raft_consensus.cc:2229] T 2ac4718f25644ef0be087a16fa69f71e P 05af95f99f7f4f789524bb34975ec19f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:40:53.918464 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 29669
I20260928 06:40:53.918579 29968 raft_consensus.cc:2258] T 2ac4718f25644ef0be087a16fa69f71e P 05af95f99f7f4f789524bb34975ec19f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:40:53.919265 29944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:40:53.918728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:46369 (local address 127.0.0.1:40564)
0928 06:40:53.919196 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 06:40:53.919236 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.65:46369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448}
W20260928 06:40:53.919442 29460 catalog_manager.cc:4584] TS 79ef270434c64b7a96b29949ee465db6 (127.18.211.65:46369): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2ac4718f25644ef0be087a16fa69f71e: Network error: Client connection negotiation failed: client connection to 127.18.211.65:46369: connect: Connection refused (error 111)
I20260928 06:40:53.920073 29968 ts_tablet_manager.cc:1905] T 2ac4718f25644ef0be087a16fa69f71e P 05af95f99f7f4f789524bb34975ec19f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:40:53.921708 29968 ts_tablet_manager.cc:1918] T 2ac4718f25644ef0be087a16fa69f71e P 05af95f99f7f4f789524bb34975ec19f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:40:53.921795 29968 log.cc:1196] T 2ac4718f25644ef0be087a16fa69f71e P 05af95f99f7f4f789524bb34975ec19f: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790577602522428-19277-0/minicluster-data/ts-2/wal/wals/2ac4718f25644ef0be087a16fa69f71e
I20260928 06:40:53.922036 29968 ts_tablet_manager.cc:1939] T 2ac4718f25644ef0be087a16fa69f71e P 05af95f99f7f4f789524bb34975ec19f: Deleting consensus metadata
I20260928 06:40:53.922417 29459 catalog_manager.cc:4841] TS 05af95f99f7f4f789524bb34975ec19f (127.18.211.67:33869): tablet 2ac4718f25644ef0be087a16fa69f71e (table test-table [id=515da95eec2b4d0db7701775fb4537d5]) successfully deleted
I20260928 06:40:53.923175 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 29803
I20260928 06:40:53.927143 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 29441
2026-09-28T06:40:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1798 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:40:53 dist-test-slave-b28r krb5kdc[29974](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:40:53 dist-test-slave-b28r krb5kdc[29974](info): set up 2 sockets
Sep 28 06:40:53 dist-test-slave-b28r krb5kdc[29974](info): commencing operation
krb5kdc: starting...
W20260928 06:40:54.876675 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:40:54 dist-test-slave-b28r krb5kdc[29974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:40:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:40:54Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.18.211.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.18.211.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.18.211.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/rangeradmin_127.18.211.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.18.211.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/rangeradmin_127.18.211.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.18.211.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.18.211.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.18.211.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/rangerlookup_127.18.211.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.18.211.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/rangerlookup_127.18.211.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.211.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.211.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.211.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/HTTP_127.18.211.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.211.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/HTTP_127.18.211.84@KRBTEST.COM.keytab.
I20260928 06:40:55.025586 19277 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskOUYCUS/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/postgres -l logfile start
2026-09-28 06:40:57.319 UTC [30004] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 06:40:57.319 UTC [30004] LOG: listening on IPv4 address "127.18.211.84", port 42895
2026-09-28 06:40:57.322 UTC [30004] LOG: listening on Unix socket "/tmp/.s.PGSQL.42895"
2026-09-28 06:40:57.349 UTC [30007] LOG: database system was shut down at 2026-09-28 06:40:56 UTC
2026-09-28 06:40:57.352 UTC [30004] LOG: database system is ready to accept connections
I20260928 06:40:58.252671 19277 mini_postgres.cc:96] Postgres bound to 42895
2026-09-28 06:40:58.264 UTC [30015] FATAL: database "slave" does not exist
127.18.211.84:42895 - accepting connections
I20260928 06:40:58.265446 19277 mini_ranger.cc:161] Starting Ranger...
I20260928 06:40:58.281790 19277 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 06:40:58.705592 19277 mini_ranger.cc:87] Created ranger Postgres database
I20260928 06:40:58.705672 19277 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ranger-admin
2026-09-28 06:40:58,962 [I] DB FLAVOR :POSTGRES
2026-09-28 06:40:58,962 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 06:40:58,963 [I] Checking connection..
2026-09-28 06:40:58,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:42895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 06:40:59,238 [I] Checking connection passed.
2026-09-28 06:40:59,239 [I] --------- Verifying version history table ---------
2026-09-28 06:40:59,239 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:42895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:40:59,483 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 06:40:59,483 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:42895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:40:59,725 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 06:40:59,726 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 06:40:59,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:42895/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 06:40:59.951 UTC [30079] WARNING: there is no transaction in progress
2026-09-28 06:40:59,957 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 06:40:59,957 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:42895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:41:00,199 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 06:41:00,199 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 06:41:00,199 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:42895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 06:41:00,451 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:42895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 06:41:00,704 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:42895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-b28r.c.gcp-upstream.internal','N') ;"
2026-09-28 06:41:00,952 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:42895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 06:41:01,199 [I] Table x_portal_user does not exist in database ranger
2026-09-28 06:41:01,199 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:42895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 06:41:01,429 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 06:41:01,429 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:42895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 06:41:01,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:42895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 06:41:01,896 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 06:41:01,897 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:42895/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 06:41:02.803 UTC [30187] WARNING: there is no transaction in progress
2026-09-28 06:41:02.819 UTC [30187] WARNING: there is no transaction in progress
2026-09-28 06:41:02.835 UTC [30187] WARNING: there is no transaction in progress
2026-09-28 06:41:02.850 UTC [30187] WARNING: there is no transaction in progress
2026-09-28 06:41:02.866 UTC [30187] WARNING: there is no transaction in progress
2026-09-28 06:41:02.883 UTC [30187] WARNING: there is no transaction in progress
2026-09-28 06:41:02.949 UTC [30187] WARNING: there is no transaction in progress
2026-09-28 06:41:02.956 UTC [30187] WARNING: there is no transaction in progress
2026-09-28 06:41:02.969 UTC [30187] WARNING: there is no transaction in progress
2026-09-28 06:41:02.979 UTC [30187] WARNING: there is no transaction in progress
2026-09-28 06:41:02.988 UTC [30187] WARNING: there is no transaction in progress
2026-09-28 06:41:03.002 UTC [30187] WARNING: there is no transaction in progress
2026-09-28 06:41:03.011 UTC [30187] WARNING: there is no transaction in progress
2026-09-28 06:41:03.018 UTC [30187] WARNING: there is no transaction in progress
2026-09-28 06:41:03.024 UTC [30187] WARNING: there is no transaction in progress
2026-09-28 06:41:03.029 UTC [30187] WARNING: there is no transaction in progress
2026-09-28 06:41:03,629 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 06:41:03,630 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:42895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 06:41:03,845 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 06:41:03,845 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:42895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 06:41:04,077 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 06:41:04,078 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:42895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 06:41:04,317 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 06:41:04,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:42895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 06:41:04,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:42895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 06:41:04,773 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:42895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-b28r.c.gcp-upstream.internal';"
2026-09-28 06:41:04,999 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 06:41:04,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:42895/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 06:41:05,253 [I] DB_PATCHES have already been applied
I20260928 06:41:05.259302 19277 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ranger-admin:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskOUYCUS/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 06:41:05.259371 19277 mini_ranger.cc:193] Using host: 127.18.211.84
I20260928 06:41:05.262372 19277 mini_ranger.cc:227] Ranger admin URL: http://127.18.211.84:56927
Sep 28, 2026 6:41:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 6:41:05 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 6:41:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 6:41:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 6:41:05 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 6:41:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 6:41:06 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 06:41:06.262022 19277 mini_ranger.cc:160] Time spent starting Ranger: real 7.997s user 0.000s sys 0.008s
Sep 28, 2026 6:41:26 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 6:41:26 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 28, 2026 6:41:26 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 6:41:26 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 6:41:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 6:41:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 6:41:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 6:41:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 6:41:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 6:41:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 6:41:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 6:41:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 6:41:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 6:41:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 6:41:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 6:41:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 6:41:26 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 6:41:27 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 06:41:28.212176 19277 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.18.211.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.18.211.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.18.211.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/rangerkms_127.18.211.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.18.211.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/rangerkms_127.18.211.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.211.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.18.211.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.211.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/HTTP_127.18.211.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.211.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/HTTP_127.18.211.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 06:41:28 dist-test-slave-b28r krb5kdc[29974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577688, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 06:41:28.287345 19277 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 06:41:28.296398 19277 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 06:41:29.069768 19277 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 06:41:29.069906 19277 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ranger-kms
I20260928 06:41:29.069919 19277 mini_ranger_kms.cc:227] Using postgres at 127.18.211.84:42895
2026-09-28 06:41:29,786 [I] DB FLAVOR :POSTGRES
2026-09-28 06:41:29,786 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 06:41:29,786 [I] Checking connection
2026-09-28 06:41:29,786 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:42895/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 06:41:30,050 [I] connection success
2026-09-28 06:41:30,050 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 06:41:30,050 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 06:41:30,050 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:42895/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 06:41:30,285 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 06:41:30,285 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 06:41:30,285 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 06:41:30,285 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:42895/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 06:41:30,559 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 06:41:31.035758 19277 mini_ranger_kms.cc:310] Created kms service
I20260928 06:41:31.207580 19277 mini_ranger_kms.cc:326] Created kudu user
I20260928 06:41:31.254431 19277 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 06:41:31 dist-test-slave-b28r krb5kdc[29974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577688, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.18.211.84@KRBTEST.COM
I20260928 06:41:31.630831 19277 mini_ranger_kms.cc:384] Added ranger policy
I20260928 06:41:31.630955 19277 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ranger-kms:/tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 06:41:31.630975 19277 mini_ranger_kms.cc:240] Using host: 127.18.211.84
I20260928 06:41:31.635353 19277 mini_ranger_kms.cc:276] Ranger KMS PID: 30367
I20260928 06:41:31.635490 19277 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.18.211.84:46385
Sep 28, 2026 6:41:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 6:41:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 6:41:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 6:41:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 6:41:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 6:41:33 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 06:41:33.793848 19277 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.507s user 0.000s sys 0.280s
I20260928 06:41:33.793957 19277 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 06:41:33.794041 19277 mini_ranger_kms.cc:401] 127.18.211.84:46385/kms/v1/keys
Sep 28 06:41:34 dist-test-slave-b28r krb5kdc[29974](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790577694, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.211.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 06:41:34 dist-test-slave-b28r krb5kdc[29974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577694, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.211.84@KRBTEST.COM for HTTP/127.18.211.84@KRBTEST.COM
Sep 28 06:41:34 dist-test-slave-b28r krb5kdc[29974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577694, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.211.84@KRBTEST.COM for HTTP/127.18.211.84@KRBTEST.COM
Sep 28 06:41:37 dist-test-slave-b28r krb5kdc[29974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.18.211.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
I20260928 06:41:37.132046 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:40069
--webserver_interface=127.18.211.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:33953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:40069
--ranger_config_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.211.84:46385/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:41:37.215523 30418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:41:37.215739 30418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:41:37.215760 30418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:41:37.216972 30418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:41:37.217015 30418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:41:37.217028 30418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:41:37.217039 30418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:41:37.217049 30418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:41:37.218660 30418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:33953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/master-0/wal
--ranger_kms_url=127.18.211.84:46385/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:40069
--ranger_config_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:40069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:41:37.218889 30418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:41:37.219135 30418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:41:37.223886 30426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:41:37.223928 30423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:41:37.223882 30424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:41:37.224416 30418 server_base.cc:1034] running on GCE node
I20260928 06:41:37.224658 30418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:41:37.224941 30418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:41:37.226089 30418 hybrid_clock.cc:648] HybridClock initialized: now 1790577697226078 us; error 37 us; skew 500 ppm
Sep 28 06:41:37 dist-test-slave-b28r krb5kdc[29974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577697, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:41:37.232586 30418 init.cc:377] Logged in from keytab as kudu/127.18.211.126@KRBTEST.COM (short username kudu)
I20260928 06:41:37.233090 30418 webserver.cc:460] Webserver started at http://127.18.211.126:43119/ using document root <none> and password file <none>
I20260928 06:41:37.233297 30418 fs_manager.cc:359] Metadata directory not provided
I20260928 06:41:37.233347 30418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:41:37.233471 30418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:41:37 dist-test-slave-b28r krb5kdc[29974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577697, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for HTTP/127.18.211.84@KRBTEST.COM
I20260928 06:41:37.288928 30418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "644d4704ec6844818a6a45b20eeda257"
format_stamp: "Formatted at 2026-09-28 06:41:37 on dist-test-slave-b28r"
server_key: "51e5f072a11c02abcac00b32303512c0"
server_key_iv: "68c1927df91981f72ec6cd702ff6ab07"
server_key_version: "kuduclusterkey@0"
I20260928 06:41:37.289376 30418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "644d4704ec6844818a6a45b20eeda257"
format_stamp: "Formatted at 2026-09-28 06:41:37 on dist-test-slave-b28r"
server_key: "51e5f072a11c02abcac00b32303512c0"
server_key_iv: "68c1927df91981f72ec6cd702ff6ab07"
server_key_version: "kuduclusterkey@0"
I20260928 06:41:37.291101 30418 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:41:37.302829 30433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:41:37.303030 30418 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:41:37.303156 30418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "644d4704ec6844818a6a45b20eeda257"
format_stamp: "Formatted at 2026-09-28 06:41:37 on dist-test-slave-b28r"
server_key: "51e5f072a11c02abcac00b32303512c0"
server_key_iv: "68c1927df91981f72ec6cd702ff6ab07"
server_key_version: "kuduclusterkey@0"
I20260928 06:41:37.303254 30418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:41:37.320449 30418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:41:37.322556 30418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:41:37.322758 30418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:41:37.328701 30418 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:40069
I20260928 06:41:37.328749 30485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:40069 every 8 connection(s)
I20260928 06:41:37.329085 30418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:41:37.329715 30486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:41:37.332134 30486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 644d4704ec6844818a6a45b20eeda257: Bootstrap starting.
I20260928 06:41:37.332760 30486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 644d4704ec6844818a6a45b20eeda257: Neither blocks nor log segments found. Creating new log.
I20260928 06:41:37.333024 30486 log.cc:824] T 00000000000000000000000000000000 P 644d4704ec6844818a6a45b20eeda257: Log is configured to *not* fsync() on all Append() calls
I20260928 06:41:37.333647 30486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 644d4704ec6844818a6a45b20eeda257: No bootstrap required, opened a new log
I20260928 06:41:37.335837 30486 raft_consensus.cc:348] T 00000000000000000000000000000000 P 644d4704ec6844818a6a45b20eeda257 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "644d4704ec6844818a6a45b20eeda257" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 40069 } }
I20260928 06:41:37.336035 30486 raft_consensus.cc:374] T 00000000000000000000000000000000 P 644d4704ec6844818a6a45b20eeda257 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:41:37.336076 30486 raft_consensus.cc:729] T 00000000000000000000000000000000 P 644d4704ec6844818a6a45b20eeda257 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 644d4704ec6844818a6a45b20eeda257, State: Initialized, Role: FOLLOWER
I20260928 06:41:37.336267 30486 consensus_queue.cc:260] T 00000000000000000000000000000000 P 644d4704ec6844818a6a45b20eeda257 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "644d4704ec6844818a6a45b20eeda257" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 40069 } }
I20260928 06:41:37.336339 30486 raft_consensus.cc:388] T 00000000000000000000000000000000 P 644d4704ec6844818a6a45b20eeda257 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:41:37.336369 30486 raft_consensus.cc:482] T 00000000000000000000000000000000 P 644d4704ec6844818a6a45b20eeda257 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:41:37.336428 30486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 644d4704ec6844818a6a45b20eeda257 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:41:37.336983 30486 raft_consensus.cc:504] T 00000000000000000000000000000000 P 644d4704ec6844818a6a45b20eeda257 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "644d4704ec6844818a6a45b20eeda257" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 40069 } }
I20260928 06:41:37.337078 30486 leader_election.cc:302] T 00000000000000000000000000000000 P 644d4704ec6844818a6a45b20eeda257 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 644d4704ec6844818a6a45b20eeda257; no voters:
I20260928 06:41:37.337265 30486 leader_election.cc:288] T 00000000000000000000000000000000 P 644d4704ec6844818a6a45b20eeda257 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:41:37.337416 30490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 644d4704ec6844818a6a45b20eeda257 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:41:37.337513 30486 sys_catalog.cc:564] T 00000000000000000000000000000000 P 644d4704ec6844818a6a45b20eeda257 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:41:37.337591 30490 raft_consensus.cc:686] T 00000000000000000000000000000000 P 644d4704ec6844818a6a45b20eeda257 [term 1 LEADER]: Becoming Leader. State: Replica: 644d4704ec6844818a6a45b20eeda257, State: Running, Role: LEADER
I20260928 06:41:37.337729 30490 consensus_queue.cc:237] T 00000000000000000000000000000000 P 644d4704ec6844818a6a45b20eeda257 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "644d4704ec6844818a6a45b20eeda257" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 40069 } }
I20260928 06:41:37.338074 30490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 644d4704ec6844818a6a45b20eeda257 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "644d4704ec6844818a6a45b20eeda257" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "644d4704ec6844818a6a45b20eeda257" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 40069 } } }
I20260928 06:41:37.338222 30490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 644d4704ec6844818a6a45b20eeda257 [sys.catalog]: This master's current role is: LEADER
I20260928 06:41:37.338403 30490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 644d4704ec6844818a6a45b20eeda257 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 644d4704ec6844818a6a45b20eeda257. Latest consensus state: current_term: 1 leader_uuid: "644d4704ec6844818a6a45b20eeda257" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "644d4704ec6844818a6a45b20eeda257" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 40069 } } }
I20260928 06:41:37.338506 30490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 644d4704ec6844818a6a45b20eeda257 [sys.catalog]: This master's current role is: LEADER
I20260928 06:41:37.338667 30498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:41:37.338994 30486 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 06:41:37.339236 30498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:41:37.339852 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 30418
I20260928 06:41:37.339972 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:41:37.340173 19277 external_mini_cluster.cc:1428] Setting key 7bcfda588b362881e0ea21181a1f38ea
I20260928 06:41:37.345211 30509 mvcc.cc:205] Tried to move back new op lower bound from 7334206248322564096 to 7334206248295870464. Current Snapshot: MvccSnapshot[applied={T|T < 7334206248322564096}]
I20260928 06:41:37.346127 30498 catalog_manager.cc:1347] Generated new cluster ID: f09d5957aa9f47459e8c43ad061a6c32
I20260928 06:41:37.346189 30498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:41:37 dist-test-slave-b28r krb5kdc[29974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577697, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:41:37.358132 30498 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:41:37.358827 30498 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:41:37.362316 30508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:41:37.341950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59506 (local address 127.18.211.126:40069)
0928 06:41:37.342569 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0928 06:41:37.342583 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 06:41:37.342654 (+ 71us) server_negotiation.cc:407] Connection header received
0928 06:41:37.343121 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:41:37.343130 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:41:37.343252 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:41:37.343436 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:41:37.347052 (+ 3616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:41:37.348882 (+ 1830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:41:37.350784 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:41:37.351497 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:41:37.351607 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:41:37.358120 (+ 6513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:41:37.358126 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:41:37.358126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:41:37.358140 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:41:37.360580 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:41:37.361009 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:41:37.361012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:41:37.361013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:41:37.361122 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:41:37.361345 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:41:37.361345 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:41:37.361346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:41:37.361660 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:41:37.361824 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 06:41:37.361929 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 06:41:37.362083 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":214,"threads_started":1}
I20260928 06:41:37.364599 30498 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 644d4704ec6844818a6a45b20eeda257: Generated new TSK 0
I20260928 06:41:37.364869 30498 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.211.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab.
Entry for principal kudu/127.18.211.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab.
I20260928 06:41:38.112111 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.211.84:46385/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.65:0
--local_ip_for_outbound_sockets=127.18.211.65
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:40069
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:33953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:41:38.224181 30531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:41:38.224375 30531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:41:38.224404 30531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:41:38.225517 30531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:41:38.225556 30531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:41:38.225590 30531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.65
I20260928 06:41:38.227025 30531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:33953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.18.211.84:46385/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:40069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.65
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:41:38.227270 30531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:41:38.227494 30531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:41:38.232311 30539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:41:38.232362 30537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:41:38.232452 30536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:41:38.232688 30531 server_base.cc:1034] running on GCE node
I20260928 06:41:38.232892 30531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:41:38.233139 30531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:41:38.234305 30531 hybrid_clock.cc:648] HybridClock initialized: now 1790577698234291 us; error 26 us; skew 500 ppm
Sep 28 06:41:38 dist-test-slave-b28r krb5kdc[29974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577698, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:41:38.237567 30531 init.cc:377] Logged in from keytab as kudu/127.18.211.65@KRBTEST.COM (short username kudu)
I20260928 06:41:38.238201 30531 webserver.cc:460] Webserver started at http://127.18.211.65:40979/ using document root <none> and password file <none>
I20260928 06:41:38.238444 30531 fs_manager.cc:359] Metadata directory not provided
I20260928 06:41:38.238514 30531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:41:38.238662 30531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:41:38 dist-test-slave-b28r krb5kdc[29974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577698, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for HTTP/127.18.211.84@KRBTEST.COM
I20260928 06:41:38.255590 30531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-0/data/instance:
uuid: "dc35b5034cd44955b9f148b99b33214d"
format_stamp: "Formatted at 2026-09-28 06:41:38 on dist-test-slave-b28r"
server_key: "d06f66e3a7c9dc7d5750a0a0897c2c56"
server_key_iv: "1ff559e59a487139154860536fd85b05"
server_key_version: "kuduclusterkey@0"
I20260928 06:41:38.256143 30531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance:
uuid: "dc35b5034cd44955b9f148b99b33214d"
format_stamp: "Formatted at 2026-09-28 06:41:38 on dist-test-slave-b28r"
server_key: "d06f66e3a7c9dc7d5750a0a0897c2c56"
server_key_iv: "1ff559e59a487139154860536fd85b05"
server_key_version: "kuduclusterkey@0"
I20260928 06:41:38.257814 30531 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:41:38.279285 30548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:41:38.279520 30531 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:41:38.279623 30531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-0/wal
uuid: "dc35b5034cd44955b9f148b99b33214d"
format_stamp: "Formatted at 2026-09-28 06:41:38 on dist-test-slave-b28r"
server_key: "d06f66e3a7c9dc7d5750a0a0897c2c56"
server_key_iv: "1ff559e59a487139154860536fd85b05"
server_key_version: "kuduclusterkey@0"
I20260928 06:41:38.279760 30531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:41:38.292207 30531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:41:38.294059 30531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:41:38.294209 30531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:41:38.294404 30531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:41:38.294757 30531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:41:38.294793 30531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:41:38.294822 30531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:41:38.294860 30531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:41:38.301796 30531 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.65:33965
I20260928 06:41:38.301879 30661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.65:33965 every 8 connection(s)
I20260928 06:41:38.302165 30531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
I20260928 06:41:38.307708 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 30531
I20260928 06:41:38.307819 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance
I20260928 06:41:38.308038 19277 external_mini_cluster.cc:1428] Setting key fa454cc98de3f6577d7a8a8aa356067c
Sep 28 06:41:38 dist-test-slave-b28r krb5kdc[29974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577698, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:41:38.313118 30665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:41:38.302978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:40735 (local address 127.18.211.126:40069)
0928 06:41:38.303138 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 06:41:38.303140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:41:38.303356 (+ 216us) server_negotiation.cc:407] Connection header received
0928 06:41:38.304049 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:41:38.304050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:41:38.304063 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:41:38.304118 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:41:38.304824 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:41:38.305329 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:41:38.306889 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:41:38.307222 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:41:38.307289 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:41:38.309860 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:41:38.309861 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:41:38.309861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:41:38.309869 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:41:38.311552 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:41:38.312016 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:41:38.312017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:41:38.312018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:41:38.312083 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:41:38.312348 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:41:38.312349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:41:38.312349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:41:38.312488 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:41:38.312547 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:41:38.312922 (+ 375us) server_negotiation.cc:299] Negotiation successful
0928 06:41:38.313014 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260928 06:41:38.313256 30664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:41:38.302911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:40069 (local address 127.18.211.65:40735)
0928 06:41:38.303209 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 06:41:38.303258 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 06:41:38.303920 (+ 662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:41:38.304199 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:41:38.304204 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:41:38.304447 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:41:38.304717 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:41:38.304724 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:41:38.305426 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:41:38.305427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:41:38.306786 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:41:38.306789 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:41:38.307360 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:41:38.307361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:41:38.307507 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:41:38.308118 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:41:38.308128 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:41:38.309753 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:41:38.311636 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:41:38.311643 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:41:38.311645 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:41:38.311943 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:41:38.312141 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:41:38.312142 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:41:38.312142 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:41:38.312283 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:41:38.312577 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:41:38.312584 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:41:38.312764 (+ 180us) client_negotiation.cc:769] Sending connection context
0928 06:41:38.312874 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 06:41:38.313024 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":117,"threads_started":1}
I20260928 06:41:38.313849 30662 heartbeater.cc:347] Connected to a master server at 127.18.211.126:40069
I20260928 06:41:38.313949 30662 heartbeater.cc:464] Registering TS with master...
I20260928 06:41:38.314137 30662 heartbeater.cc:511] Master 127.18.211.126:40069 requested a full tablet report, sending...
I20260928 06:41:38.315506 30450 ts_manager.cc:194] Registered new tserver with Master: dc35b5034cd44955b9f148b99b33214d (127.18.211.65:33965)
I20260928 06:41:38.316390 30450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.65@KRBTEST.COM'} at 127.18.211.65:40735
WARNING: no policy specified for kudu/127.18.211.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab.
Entry for principal kudu/127.18.211.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab.
I20260928 06:41:38.339370 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.211.84:46385/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.66:0
--local_ip_for_outbound_sockets=127.18.211.66
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:40069
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:33953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:41:38.437141 30668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:41:38.437400 30668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:41:38.437429 30668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:41:38.439034 30668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:41:38.439102 30668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:41:38.439141 30668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.66
I20260928 06:41:38.441246 30668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:33953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.18.211.84:46385/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:40069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.66
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:41:38.441522 30668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:41:38.441802 30668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:41:38.446830 30676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:41:38.446830 30678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:41:38.447393 30675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:41:38.447495 30668 server_base.cc:1034] running on GCE node
I20260928 06:41:38.447703 30668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:41:38.447935 30668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:41:38.449084 30668 hybrid_clock.cc:648] HybridClock initialized: now 1790577698449068 us; error 35 us; skew 500 ppm
Sep 28 06:41:38 dist-test-slave-b28r krb5kdc[29974](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790577698, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 06:41:38 dist-test-slave-b28r krb5kdc[29974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577698, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:41:38.452044 30668 init.cc:377] Logged in from keytab as kudu/127.18.211.66@KRBTEST.COM (short username kudu)
I20260928 06:41:38.452483 30668 webserver.cc:460] Webserver started at http://127.18.211.66:34987/ using document root <none> and password file <none>
I20260928 06:41:38.452711 30668 fs_manager.cc:359] Metadata directory not provided
I20260928 06:41:38.452759 30668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:41:38.452879 30668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:41:38 dist-test-slave-b28r krb5kdc[29974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577698, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for HTTP/127.18.211.84@KRBTEST.COM
I20260928 06:41:38.465229 30668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-1/data/instance:
uuid: "621d11136b6344809ce46c7d56ef6df7"
format_stamp: "Formatted at 2026-09-28 06:41:38 on dist-test-slave-b28r"
server_key: "76bb9529ddd6dc228142bff79ebe401b"
server_key_iv: "0bf5fa0aa3573e52795cf6295def1614"
server_key_version: "kuduclusterkey@0"
I20260928 06:41:38.465708 30668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance:
uuid: "621d11136b6344809ce46c7d56ef6df7"
format_stamp: "Formatted at 2026-09-28 06:41:38 on dist-test-slave-b28r"
server_key: "76bb9529ddd6dc228142bff79ebe401b"
server_key_iv: "0bf5fa0aa3573e52795cf6295def1614"
server_key_version: "kuduclusterkey@0"
I20260928 06:41:38.467466 30668 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260928 06:41:38.491242 30685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:41:38.491523 30668 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.000s sys 0.000s
I20260928 06:41:38.491629 30668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-1/wal
uuid: "621d11136b6344809ce46c7d56ef6df7"
format_stamp: "Formatted at 2026-09-28 06:41:38 on dist-test-slave-b28r"
server_key: "76bb9529ddd6dc228142bff79ebe401b"
server_key_iv: "0bf5fa0aa3573e52795cf6295def1614"
server_key_version: "kuduclusterkey@0"
I20260928 06:41:38.491734 30668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:41:38.506517 30668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:41:38.508785 30668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:41:38.508937 30668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:41:38.509140 30668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:41:38.509514 30668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:41:38.509555 30668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:41:38.509583 30668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:41:38.509634 30668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:41:38.517110 30668 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.66:41785
I20260928 06:41:38.517661 30668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
I20260928 06:41:38.518577 30798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.66:41785 every 8 connection(s)
Sep 28 06:41:38 dist-test-slave-b28r krb5kdc[29974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577698, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:41:38.526684 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 30668
I20260928 06:41:38.526784 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance
I20260928 06:41:38.526955 19277 external_mini_cluster.cc:1428] Setting key 5c91bf03f7fcf608ab6895ddb4946a31
I20260928 06:41:38.529803 30665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:41:38.518502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:34325 (local address 127.18.211.126:40069)
0928 06:41:38.518555 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:41:38.518557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:41:38.519381 (+ 824us) server_negotiation.cc:407] Connection header received
0928 06:41:38.520187 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:41:38.520188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:41:38.520205 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:41:38.520280 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:41:38.521015 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:41:38.521800 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:41:38.523338 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:41:38.523651 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:41:38.523732 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:41:38.526314 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:41:38.526316 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:41:38.526316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:41:38.526324 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:41:38.528240 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:41:38.528706 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:41:38.528708 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:41:38.528708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:41:38.528778 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:41:38.529033 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:41:38.529034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:41:38.529034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:41:38.529158 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:41:38.529196 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 06:41:38.529599 (+ 403us) server_negotiation.cc:299] Negotiation successful
0928 06:41:38.529696 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:41:38.530426 30801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:41:38.518513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:40069 (local address 127.18.211.66:34325)
0928 06:41:38.519232 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0928 06:41:38.519278 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 06:41:38.520063 (+ 785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:41:38.520374 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:41:38.520379 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:41:38.520612 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:41:38.520902 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:41:38.520909 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:41:38.521906 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:41:38.521907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:41:38.523246 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:41:38.523249 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:41:38.523756 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:41:38.523757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:41:38.523905 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:41:38.524522 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:41:38.524532 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:41:38.526199 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:41:38.528329 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:41:38.528337 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:41:38.528340 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:41:38.528628 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:41:38.528840 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:41:38.528841 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:41:38.528841 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:41:38.528971 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:41:38.529240 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:41:38.529247 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:41:38.529433 (+ 186us) client_negotiation.cc:769] Sending connection context
0928 06:41:38.530019 (+ 586us) client_negotiation.cc:240] Negotiation successful
0928 06:41:38.530191 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":202,"threads_started":1}
I20260928 06:41:38.530975 30799 heartbeater.cc:347] Connected to a master server at 127.18.211.126:40069
I20260928 06:41:38.531121 30799 heartbeater.cc:464] Registering TS with master...
I20260928 06:41:38.531333 30799 heartbeater.cc:511] Master 127.18.211.126:40069 requested a full tablet report, sending...
I20260928 06:41:38.531723 30450 ts_manager.cc:194] Registered new tserver with Master: 621d11136b6344809ce46c7d56ef6df7 (127.18.211.66:41785)
I20260928 06:41:38.532188 30450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.66@KRBTEST.COM'} at 127.18.211.66:34325
WARNING: no policy specified for kudu/127.18.211.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab.
Entry for principal kudu/127.18.211.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab.
I20260928 06:41:38.565657 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.211.84:46385/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.67:0
--local_ip_for_outbound_sockets=127.18.211.67
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:40069
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:33953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:41:38.699326 30805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:41:38.699671 30805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:41:38.699736 30805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:41:38.701429 30805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:41:38.701530 30805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:41:38.701597 30805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.67
I20260928 06:41:38.703958 30805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:33953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.18.211.84:46385/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:40069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.67
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:41:38.704342 30805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:41:38.704687 30805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:41:38.711776 30813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:41:38.716077 30811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:41:38.716295 30810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:41:38.720355 30805 server_base.cc:1034] running on GCE node
I20260928 06:41:38.720726 30805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:41:38.721047 30805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:41:38.722925 30805 hybrid_clock.cc:648] HybridClock initialized: now 1790577698722904 us; error 36 us; skew 500 ppm
Sep 28 06:41:38 dist-test-slave-b28r krb5kdc[29974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577698, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:41:38.729512 30805 init.cc:377] Logged in from keytab as kudu/127.18.211.67@KRBTEST.COM (short username kudu)
I20260928 06:41:38.730060 30805 webserver.cc:460] Webserver started at http://127.18.211.67:40371/ using document root <none> and password file <none>
I20260928 06:41:38.730319 30805 fs_manager.cc:359] Metadata directory not provided
I20260928 06:41:38.730367 30805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:41:38.730490 30805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:41:38 dist-test-slave-b28r krb5kdc[29974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577698, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for HTTP/127.18.211.84@KRBTEST.COM
I20260928 06:41:38.758302 30805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-2/data/instance:
uuid: "776c2903f13e4fdfbbd84a818712b42a"
format_stamp: "Formatted at 2026-09-28 06:41:38 on dist-test-slave-b28r"
server_key: "82e6f24e49eb958514d5c178147fcf05"
server_key_iv: "2dc065510bfc2230219c2866978ae0a2"
server_key_version: "kuduclusterkey@0"
I20260928 06:41:38.758875 30805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance:
uuid: "776c2903f13e4fdfbbd84a818712b42a"
format_stamp: "Formatted at 2026-09-28 06:41:38 on dist-test-slave-b28r"
server_key: "82e6f24e49eb958514d5c178147fcf05"
server_key_iv: "2dc065510bfc2230219c2866978ae0a2"
server_key_version: "kuduclusterkey@0"
I20260928 06:41:38.766199 30805 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.001s
I20260928 06:41:38.779222 30820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:41:38.779405 30805 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:41:38.779493 30805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-2/wal
uuid: "776c2903f13e4fdfbbd84a818712b42a"
format_stamp: "Formatted at 2026-09-28 06:41:38 on dist-test-slave-b28r"
server_key: "82e6f24e49eb958514d5c178147fcf05"
server_key_iv: "2dc065510bfc2230219c2866978ae0a2"
server_key_version: "kuduclusterkey@0"
I20260928 06:41:38.779593 30805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:41:38.814643 30805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:41:38.817294 30805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:41:38.817517 30805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:41:38.817839 30805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:41:38.818315 30805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:41:38.818403 30805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:41:38.818470 30805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:41:38.818522 30805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:41:38.826815 30805 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.67:41029
I20260928 06:41:38.827543 30805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
I20260928 06:41:38.836895 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 30805
I20260928 06:41:38.837008 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance
I20260928 06:41:38.837178 19277 external_mini_cluster.cc:1428] Setting key a8ccd86463c1bfaf3effeb523e55e52f
I20260928 06:41:38.830132 30933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.67:41029 every 8 connection(s)
Sep 28 06:41:38 dist-test-slave-b28r krb5kdc[29974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577698, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:41:38.842339 30665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:41:38.830975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:52945 (local address 127.18.211.126:40069)
0928 06:41:38.831027 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:41:38.831029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:41:38.831267 (+ 238us) server_negotiation.cc:407] Connection header received
0928 06:41:38.832023 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:41:38.832025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:41:38.832038 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:41:38.832070 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:41:38.832967 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:41:38.833674 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:41:38.835264 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:41:38.835558 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:41:38.835636 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:41:38.839026 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:41:38.839027 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:41:38.839028 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:41:38.839036 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:41:38.840677 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:41:38.841183 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:41:38.841184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:41:38.841184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:41:38.841249 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:41:38.841557 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:41:38.841558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:41:38.841558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:41:38.841701 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:41:38.841759 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:41:38.842153 (+ 394us) server_negotiation.cc:299] Negotiation successful
0928 06:41:38.842240 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:41:38.843124 30936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:41:38.830728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:40069 (local address 127.18.211.67:52945)
0928 06:41:38.830948 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 06:41:38.831069 (+ 121us) client_negotiation.cc:174] Beginning negotiation
0928 06:41:38.831884 (+ 815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:41:38.832284 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:41:38.832289 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:41:38.832564 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:41:38.832859 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:41:38.832866 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:41:38.833780 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:41:38.833781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:41:38.835166 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:41:38.835169 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:41:38.835657 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:41:38.835658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:41:38.835840 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:41:38.836576 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:41:38.836587 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:41:38.838914 (+ 2327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:41:38.840785 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:41:38.840792 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:41:38.840794 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:41:38.841089 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:41:38.841324 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:41:38.841325 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:41:38.841326 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:41:38.841473 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:41:38.841809 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:41:38.841816 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:41:38.841986 (+ 170us) client_negotiation.cc:769] Sending connection context
0928 06:41:38.842662 (+ 676us) client_negotiation.cc:240] Negotiation successful
0928 06:41:38.842845 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":89,"threads_started":1}
I20260928 06:41:38.843981 30934 heartbeater.cc:347] Connected to a master server at 127.18.211.126:40069
I20260928 06:41:38.844135 30934 heartbeater.cc:464] Registering TS with master...
I20260928 06:41:38.844395 30934 heartbeater.cc:511] Master 127.18.211.126:40069 requested a full tablet report, sending...
I20260928 06:41:38.844810 30450 ts_manager.cc:194] Registered new tserver with Master: 776c2903f13e4fdfbbd84a818712b42a (127.18.211.67:41029)
I20260928 06:41:38.845299 30450 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.67@KRBTEST.COM'} at 127.18.211.67:52945
I20260928 06:41:38.849841 19277 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:41:38.867928 30665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:41:38.851436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34686 (local address 127.18.211.126:40069)
0928 06:41:38.851477 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:41:38.851479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:41:38.851820 (+ 341us) server_negotiation.cc:407] Connection header received
0928 06:41:38.851919 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:41:38.851920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:41:38.851930 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:41:38.851959 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:41:38.852779 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:41:38.853552 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:41:38.854683 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:41:38.854995 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:41:38.855036 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:41:38.856703 (+ 1667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:41:38.856704 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:41:38.856705 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:41:38.856712 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:41:38.858337 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:41:38.858753 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:41:38.858754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:41:38.858754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:41:38.858815 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:41:38.859014 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:41:38.859014 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:41:38.859015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:41:38.867319 (+ 8304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:41:38.867431 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 06:41:38.867690 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 06:41:38.867801 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:41:38.870393 30450 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34686:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:41:38.872772 30450 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:41:38.886196 30951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:41:38.876391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:33965 (local address 127.0.0.1:36716)
0928 06:41:38.877824 (+ 1433us) negotiation.cc:107] Waiting for socket to connect
0928 06:41:38.877867 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:41:38.877939 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:41:38.878332 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:41:38.878335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:41:38.878338 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:41:38.878420 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:41:38.878423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:41:38.882330 (+ 3907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:41:38.882331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:41:38.884713 (+ 2382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:41:38.884716 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:41:38.885871 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:41:38.885872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:41:38.885961 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:41:38.885978 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 06:41:38.886052 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 06:41:38.886107 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1405,"thread_start_us":89,"threads_started":1}
I20260928 06:41:38.888810 30947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:41:38.877129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:41785 (local address 127.0.0.1:33652)
0928 06:41:38.879729 (+ 2600us) negotiation.cc:107] Waiting for socket to connect
0928 06:41:38.879747 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 06:41:38.879787 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:41:38.880044 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:41:38.880045 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:41:38.880047 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:41:38.880088 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:41:38.880089 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:41:38.881324 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:41:38.881325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:41:38.886803 (+ 5478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:41:38.886807 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:41:38.888547 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:41:38.888548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:41:38.888615 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:41:38.888619 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:41:38.888668 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 06:41:38.888709 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2588,"thread_start_us":68,"threads_started":1}
I20260928 06:41:38.889277 30950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:41:38.876849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36716 (local address 127.18.211.65:33965)
0928 06:41:38.877755 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0928 06:41:38.877761 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:41:38.878107 (+ 346us) server_negotiation.cc:407] Connection header received
0928 06:41:38.878145 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:41:38.878147 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:41:38.878163 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:41:38.878234 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:41:38.878508 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:41:38.882235 (+ 3727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:41:38.884887 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:41:38.885681 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:41:38.885765 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:41:38.885834 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 06:41:38.889143 (+ 3309us) server_negotiation.cc:299] Negotiation successful
0928 06:41:38.889196 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":854,"thread_start_us":59,"threads_started":1}
I20260928 06:41:38.889614 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:41:38.877378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33652 (local address 127.18.211.66:41785)
0928 06:41:38.878080 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0928 06:41:38.878085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:41:38.879839 (+ 1754us) server_negotiation.cc:407] Connection header received
0928 06:41:38.879883 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:41:38.879885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:41:38.879900 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:41:38.879967 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:41:38.880162 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:41:38.881228 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:41:38.886926 (+ 5698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:41:38.888366 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:41:38.888432 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:41:38.888513 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 06:41:38.889496 (+ 983us) server_negotiation.cc:299] Negotiation successful
0928 06:41:38.889538 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":79,"threads_started":1}
I20260928 06:41:38.890630 30596 tablet_service.cc:1467] Processing CreateTablet for tablet 038e326cc61c4028982856ee90904f00 (DEFAULT_TABLE table=test-table [id=d11c315fe75a4828854313f436a69e4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:41:38.891028 30596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 038e326cc61c4028982856ee90904f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:41:38.895913 30715 tablet_service.cc:1467] Processing CreateTablet for tablet 038e326cc61c4028982856ee90904f00 (DEFAULT_TABLE table=test-table [id=d11c315fe75a4828854313f436a69e4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:41:38.896288 30715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 038e326cc61c4028982856ee90904f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:41:38.900892 30953 tablet_bootstrap.cc:492] T 038e326cc61c4028982856ee90904f00 P dc35b5034cd44955b9f148b99b33214d: Bootstrap starting.
I20260928 06:41:38.901837 30953 tablet_bootstrap.cc:654] T 038e326cc61c4028982856ee90904f00 P dc35b5034cd44955b9f148b99b33214d: Neither blocks nor log segments found. Creating new log.
I20260928 06:41:38.902201 30953 log.cc:824] T 038e326cc61c4028982856ee90904f00 P dc35b5034cd44955b9f148b99b33214d: Log is configured to *not* fsync() on all Append() calls
I20260928 06:41:38.904634 30948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:41:38.876638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:41029 (local address 127.0.0.1:49926)
0928 06:41:38.879598 (+ 2960us) negotiation.cc:107] Waiting for socket to connect
0928 06:41:38.879608 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:41:38.879656 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:41:38.900435 (+ 20779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:41:38.900436 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:41:38.900439 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:41:38.900499 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:41:38.900502 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:41:38.902052 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:41:38.902054 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:41:38.903364 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:41:38.903368 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:41:38.904396 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:41:38.904398 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:41:38.904468 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:41:38.904472 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:41:38.904501 (+ 29us) client_negotiation.cc:240] Negotiation successful
0928 06:41:38.904549 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2943,"thread_start_us":63,"threads_started":1}
I20260928 06:41:38.905277 30949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:41:38.876772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49926 (local address 127.18.211.67:41029)
0928 06:41:38.899924 (+ 23152us) server_negotiation.cc:206] Beginning negotiation
0928 06:41:38.899929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:41:38.899940 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:41:38.899979 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:41:38.899981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:41:38.899998 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:41:38.900577 (+ 579us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:41:38.900663 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:41:38.901853 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:41:38.903495 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:41:38.904289 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:41:38.905018 (+ 729us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:41:38.905109 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 06:41:38.905137 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 06:41:38.905176 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23107,"thread_start_us":104,"threads_started":1}
I20260928 06:41:38.906248 30868 tablet_service.cc:1467] Processing CreateTablet for tablet 038e326cc61c4028982856ee90904f00 (DEFAULT_TABLE table=test-table [id=d11c315fe75a4828854313f436a69e4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:41:38.906682 30868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 038e326cc61c4028982856ee90904f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:41:38.909304 30954 tablet_bootstrap.cc:492] T 038e326cc61c4028982856ee90904f00 P 621d11136b6344809ce46c7d56ef6df7: Bootstrap starting.
I20260928 06:41:38.910459 30954 tablet_bootstrap.cc:654] T 038e326cc61c4028982856ee90904f00 P 621d11136b6344809ce46c7d56ef6df7: Neither blocks nor log segments found. Creating new log.
I20260928 06:41:38.910805 30954 log.cc:824] T 038e326cc61c4028982856ee90904f00 P 621d11136b6344809ce46c7d56ef6df7: Log is configured to *not* fsync() on all Append() calls
I20260928 06:41:38.911134 30956 tablet_bootstrap.cc:492] T 038e326cc61c4028982856ee90904f00 P 776c2903f13e4fdfbbd84a818712b42a: Bootstrap starting.
I20260928 06:41:38.912297 30956 tablet_bootstrap.cc:654] T 038e326cc61c4028982856ee90904f00 P 776c2903f13e4fdfbbd84a818712b42a: Neither blocks nor log segments found. Creating new log.
I20260928 06:41:38.912716 30956 log.cc:824] T 038e326cc61c4028982856ee90904f00 P 776c2903f13e4fdfbbd84a818712b42a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:41:38.912990 30953 tablet_bootstrap.cc:492] T 038e326cc61c4028982856ee90904f00 P dc35b5034cd44955b9f148b99b33214d: No bootstrap required, opened a new log
I20260928 06:41:38.913066 30953 ts_tablet_manager.cc:1397] T 038e326cc61c4028982856ee90904f00 P dc35b5034cd44955b9f148b99b33214d: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260928 06:41:38.916101 30953 raft_consensus.cc:348] T 038e326cc61c4028982856ee90904f00 P dc35b5034cd44955b9f148b99b33214d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc35b5034cd44955b9f148b99b33214d" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 33965 } } peers { permanent_uuid: "776c2903f13e4fdfbbd84a818712b42a" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 41029 } } peers { permanent_uuid: "621d11136b6344809ce46c7d56ef6df7" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 41785 } }
I20260928 06:41:38.916249 30953 raft_consensus.cc:374] T 038e326cc61c4028982856ee90904f00 P dc35b5034cd44955b9f148b99b33214d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:41:38.916283 30953 raft_consensus.cc:729] T 038e326cc61c4028982856ee90904f00 P dc35b5034cd44955b9f148b99b33214d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc35b5034cd44955b9f148b99b33214d, State: Initialized, Role: FOLLOWER
I20260928 06:41:38.916399 30953 consensus_queue.cc:260] T 038e326cc61c4028982856ee90904f00 P dc35b5034cd44955b9f148b99b33214d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc35b5034cd44955b9f148b99b33214d" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 33965 } } peers { permanent_uuid: "776c2903f13e4fdfbbd84a818712b42a" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 41029 } } peers { permanent_uuid: "621d11136b6344809ce46c7d56ef6df7" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 41785 } }
I20260928 06:41:38.916690 30953 ts_tablet_manager.cc:1428] T 038e326cc61c4028982856ee90904f00 P dc35b5034cd44955b9f148b99b33214d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260928 06:41:38.916693 30662 heartbeater.cc:503] Master 127.18.211.126:40069 was elected leader, sending a full tablet report...
I20260928 06:41:38.920503 30954 tablet_bootstrap.cc:492] T 038e326cc61c4028982856ee90904f00 P 621d11136b6344809ce46c7d56ef6df7: No bootstrap required, opened a new log
I20260928 06:41:38.920584 30954 ts_tablet_manager.cc:1397] T 038e326cc61c4028982856ee90904f00 P 621d11136b6344809ce46c7d56ef6df7: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260928 06:41:38.922566 30954 raft_consensus.cc:348] T 038e326cc61c4028982856ee90904f00 P 621d11136b6344809ce46c7d56ef6df7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc35b5034cd44955b9f148b99b33214d" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 33965 } } peers { permanent_uuid: "776c2903f13e4fdfbbd84a818712b42a" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 41029 } } peers { permanent_uuid: "621d11136b6344809ce46c7d56ef6df7" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 41785 } }
I20260928 06:41:38.922732 30954 raft_consensus.cc:374] T 038e326cc61c4028982856ee90904f00 P 621d11136b6344809ce46c7d56ef6df7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:41:38.922760 30954 raft_consensus.cc:729] T 038e326cc61c4028982856ee90904f00 P 621d11136b6344809ce46c7d56ef6df7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 621d11136b6344809ce46c7d56ef6df7, State: Initialized, Role: FOLLOWER
I20260928 06:41:38.922906 30954 consensus_queue.cc:260] T 038e326cc61c4028982856ee90904f00 P 621d11136b6344809ce46c7d56ef6df7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc35b5034cd44955b9f148b99b33214d" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 33965 } } peers { permanent_uuid: "776c2903f13e4fdfbbd84a818712b42a" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 41029 } } peers { permanent_uuid: "621d11136b6344809ce46c7d56ef6df7" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 41785 } }
I20260928 06:41:38.923183 30954 ts_tablet_manager.cc:1428] T 038e326cc61c4028982856ee90904f00 P 621d11136b6344809ce46c7d56ef6df7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:41:38.923343 30799 heartbeater.cc:503] Master 127.18.211.126:40069 was elected leader, sending a full tablet report...
I20260928 06:41:38.924621 30956 tablet_bootstrap.cc:492] T 038e326cc61c4028982856ee90904f00 P 776c2903f13e4fdfbbd84a818712b42a: No bootstrap required, opened a new log
I20260928 06:41:38.924726 30956 ts_tablet_manager.cc:1397] T 038e326cc61c4028982856ee90904f00 P 776c2903f13e4fdfbbd84a818712b42a: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20260928 06:41:38.927057 30956 raft_consensus.cc:348] T 038e326cc61c4028982856ee90904f00 P 776c2903f13e4fdfbbd84a818712b42a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc35b5034cd44955b9f148b99b33214d" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 33965 } } peers { permanent_uuid: "776c2903f13e4fdfbbd84a818712b42a" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 41029 } } peers { permanent_uuid: "621d11136b6344809ce46c7d56ef6df7" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 41785 } }
I20260928 06:41:38.927285 30956 raft_consensus.cc:374] T 038e326cc61c4028982856ee90904f00 P 776c2903f13e4fdfbbd84a818712b42a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:41:38.927325 30956 raft_consensus.cc:729] T 038e326cc61c4028982856ee90904f00 P 776c2903f13e4fdfbbd84a818712b42a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 776c2903f13e4fdfbbd84a818712b42a, State: Initialized, Role: FOLLOWER
I20260928 06:41:38.927467 30956 consensus_queue.cc:260] T 038e326cc61c4028982856ee90904f00 P 776c2903f13e4fdfbbd84a818712b42a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc35b5034cd44955b9f148b99b33214d" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 33965 } } peers { permanent_uuid: "776c2903f13e4fdfbbd84a818712b42a" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 41029 } } peers { permanent_uuid: "621d11136b6344809ce46c7d56ef6df7" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 41785 } }
I20260928 06:41:38.927740 30956 ts_tablet_manager.cc:1428] T 038e326cc61c4028982856ee90904f00 P 776c2903f13e4fdfbbd84a818712b42a: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260928 06:41:38.927855 30934 heartbeater.cc:503] Master 127.18.211.126:40069 was elected leader, sending a full tablet report...
Sep 28 06:41:39 dist-test-slave-b28r krb5kdc[29974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577698, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for HTTP/127.18.211.84@KRBTEST.COM
W20260928 06:41:39.018715 30800 tablet.cc:2369] T 038e326cc61c4028982856ee90904f00 P 621d11136b6344809ce46c7d56ef6df7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:41:39.060922 30663 tablet.cc:2369] T 038e326cc61c4028982856ee90904f00 P dc35b5034cd44955b9f148b99b33214d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:41:39.067034 30961 raft_consensus.cc:482] T 038e326cc61c4028982856ee90904f00 P 776c2903f13e4fdfbbd84a818712b42a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:41:39.067225 30961 raft_consensus.cc:504] T 038e326cc61c4028982856ee90904f00 P 776c2903f13e4fdfbbd84a818712b42a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc35b5034cd44955b9f148b99b33214d" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 33965 } } peers { permanent_uuid: "776c2903f13e4fdfbbd84a818712b42a" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 41029 } } peers { permanent_uuid: "621d11136b6344809ce46c7d56ef6df7" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 41785 } }
I20260928 06:41:39.067667 30961 leader_election.cc:288] T 038e326cc61c4028982856ee90904f00 P 776c2903f13e4fdfbbd84a818712b42a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc35b5034cd44955b9f148b99b33214d (127.18.211.65:33965), 621d11136b6344809ce46c7d56ef6df7 (127.18.211.66:41785)
I20260928 06:41:39.071846 30936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:41:39.067861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:33965 (local address 127.18.211.67:57129)
0928 06:41:39.068000 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 06:41:39.068012 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:41:39.068063 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:41:39.068622 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:41:39.068623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:41:39.068625 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:41:39.068714 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:41:39.068716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:41:39.069603 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:41:39.069604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:41:39.070789 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:41:39.070792 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:41:39.071566 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:41:39.071567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:41:39.071652 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:41:39.071656 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:41:39.071706 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 06:41:39.071748 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260928 06:41:39.072191 30950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:41:39.068334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:57129 (local address 127.18.211.65:33965)
0928 06:41:39.068395 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 06:41:39.068397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:41:39.068406 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:41:39.068435 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:41:39.068436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:41:39.068446 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:41:39.068500 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:41:39.068802 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:41:39.069479 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:41:39.070933 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:41:39.071481 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:41:39.071549 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:41:39.071557 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:41:39.072081 (+ 524us) server_negotiation.cc:299] Negotiation successful
0928 06:41:39.072127 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:41:39.072489 30616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038e326cc61c4028982856ee90904f00" candidate_uuid: "776c2903f13e4fdfbbd84a818712b42a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc35b5034cd44955b9f148b99b33214d" is_pre_election: true
I20260928 06:41:39.072677 30616 raft_consensus.cc:2454] T 038e326cc61c4028982856ee90904f00 P dc35b5034cd44955b9f148b99b33214d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 776c2903f13e4fdfbbd84a818712b42a in term 0.
I20260928 06:41:39.072962 30821 leader_election.cc:302] T 038e326cc61c4028982856ee90904f00 P 776c2903f13e4fdfbbd84a818712b42a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 776c2903f13e4fdfbbd84a818712b42a, dc35b5034cd44955b9f148b99b33214d; no voters:
I20260928 06:41:39.073112 30961 raft_consensus.cc:2781] T 038e326cc61c4028982856ee90904f00 P 776c2903f13e4fdfbbd84a818712b42a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:41:39.073164 30961 raft_consensus.cc:482] T 038e326cc61c4028982856ee90904f00 P 776c2903f13e4fdfbbd84a818712b42a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:41:39.073189 30961 raft_consensus.cc:3037] T 038e326cc61c4028982856ee90904f00 P 776c2903f13e4fdfbbd84a818712b42a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:41:39.074164 30961 raft_consensus.cc:504] T 038e326cc61c4028982856ee90904f00 P 776c2903f13e4fdfbbd84a818712b42a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc35b5034cd44955b9f148b99b33214d" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 33965 } } peers { permanent_uuid: "776c2903f13e4fdfbbd84a818712b42a" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 41029 } } peers { permanent_uuid: "621d11136b6344809ce46c7d56ef6df7" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 41785 } }
I20260928 06:41:39.074324 30961 leader_election.cc:288] T 038e326cc61c4028982856ee90904f00 P 776c2903f13e4fdfbbd84a818712b42a [CANDIDATE]: Term 1 election: Requested vote from peers dc35b5034cd44955b9f148b99b33214d (127.18.211.65:33965), 621d11136b6344809ce46c7d56ef6df7 (127.18.211.66:41785)
I20260928 06:41:39.074514 30616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038e326cc61c4028982856ee90904f00" candidate_uuid: "776c2903f13e4fdfbbd84a818712b42a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc35b5034cd44955b9f148b99b33214d"
I20260928 06:41:39.074594 30616 raft_consensus.cc:3037] T 038e326cc61c4028982856ee90904f00 P dc35b5034cd44955b9f148b99b33214d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:41:39.075757 30616 raft_consensus.cc:2454] T 038e326cc61c4028982856ee90904f00 P dc35b5034cd44955b9f148b99b33214d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 776c2903f13e4fdfbbd84a818712b42a in term 1.
I20260928 06:41:39.075938 30821 leader_election.cc:302] T 038e326cc61c4028982856ee90904f00 P 776c2903f13e4fdfbbd84a818712b42a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 776c2903f13e4fdfbbd84a818712b42a, dc35b5034cd44955b9f148b99b33214d; no voters:
I20260928 06:41:39.076009 30961 raft_consensus.cc:2781] T 038e326cc61c4028982856ee90904f00 P 776c2903f13e4fdfbbd84a818712b42a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:41:39.076123 30961 raft_consensus.cc:686] T 038e326cc61c4028982856ee90904f00 P 776c2903f13e4fdfbbd84a818712b42a [term 1 LEADER]: Becoming Leader. State: Replica: 776c2903f13e4fdfbbd84a818712b42a, State: Running, Role: LEADER
I20260928 06:41:39.076221 30961 consensus_queue.cc:237] T 038e326cc61c4028982856ee90904f00 P 776c2903f13e4fdfbbd84a818712b42a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc35b5034cd44955b9f148b99b33214d" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 33965 } } peers { permanent_uuid: "776c2903f13e4fdfbbd84a818712b42a" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 41029 } } peers { permanent_uuid: "621d11136b6344809ce46c7d56ef6df7" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 41785 } }
I20260928 06:41:39.077234 30450 catalog_manager.cc:5495] T 038e326cc61c4028982856ee90904f00 P 776c2903f13e4fdfbbd84a818712b42a reported cstate change: term changed from 0 to 1, leader changed from <none> to 776c2903f13e4fdfbbd84a818712b42a (127.18.211.67). New cstate: current_term: 1 leader_uuid: "776c2903f13e4fdfbbd84a818712b42a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc35b5034cd44955b9f148b99b33214d" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 33965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "776c2903f13e4fdfbbd84a818712b42a" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 41029 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "621d11136b6344809ce46c7d56ef6df7" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 41785 } health_report { overall_health: UNKNOWN } } }
W20260928 06:41:39.081313 30935 tablet.cc:2369] T 038e326cc61c4028982856ee90904f00 P 776c2903f13e4fdfbbd84a818712b42a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:41:39.083465 30952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:41:39.068119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:39153 (local address 127.18.211.66:41785)
0928 06:41:39.068163 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:41:39.068165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:41:39.068317 (+ 152us) server_negotiation.cc:407] Connection header received
0928 06:41:39.068415 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:41:39.068415 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:41:39.068425 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:41:39.068453 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:41:39.068661 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:41:39.069389 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:41:39.070913 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:41:39.071448 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:41:39.071495 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:41:39.071504 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:41:39.083323 (+ 11819us) server_negotiation.cc:299] Negotiation successful
0928 06:41:39.083389 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:41:39.083743 30962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:41:39.067904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:41785 (local address 127.18.211.67:39153)
0928 06:41:39.068251 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0928 06:41:39.068261 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:41:39.068342 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:41:39.068517 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:41:39.068518 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:41:39.068520 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:41:39.068595 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:41:39.068597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:41:39.069489 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:41:39.069490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:41:39.070805 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:41:39.070807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:41:39.071560 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:41:39.071560 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:41:39.083190 (+ 11630us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:41:39.083198 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 06:41:39.083591 (+ 393us) client_negotiation.cc:240] Negotiation successful
0928 06:41:39.083646 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":6,"thread_start_us":107,"threads_started":1}
I20260928 06:41:39.084084 30753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038e326cc61c4028982856ee90904f00" candidate_uuid: "776c2903f13e4fdfbbd84a818712b42a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "621d11136b6344809ce46c7d56ef6df7" is_pre_election: true
I20260928 06:41:39.084249 30753 raft_consensus.cc:2454] T 038e326cc61c4028982856ee90904f00 P 621d11136b6344809ce46c7d56ef6df7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 776c2903f13e4fdfbbd84a818712b42a in term 0.
I20260928 06:41:39.084363 30752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038e326cc61c4028982856ee90904f00" candidate_uuid: "776c2903f13e4fdfbbd84a818712b42a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "621d11136b6344809ce46c7d56ef6df7"
I20260928 06:41:39.084420 30752 raft_consensus.cc:3037] T 038e326cc61c4028982856ee90904f00 P 621d11136b6344809ce46c7d56ef6df7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:41:39.085341 30752 raft_consensus.cc:2454] T 038e326cc61c4028982856ee90904f00 P 621d11136b6344809ce46c7d56ef6df7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 776c2903f13e4fdfbbd84a818712b42a in term 1.
I20260928 06:41:39.112674 30949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:41:39.109602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49938 (local address 127.18.211.67:41029)
0928 06:41:39.109644 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:41:39.109646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:41:39.109656 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:41:39.109699 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:41:39.109700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:41:39.109724 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:41:39.109812 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:41:39.110004 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:41:39.110733 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:41:39.111888 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:41:39.112110 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:41:39.112169 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:41:39.112345 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:41:39.112458 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:41:39.112533 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:41:39.112571 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 06:41:39.112592 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:41:39.114564 30752 raft_consensus.cc:1261] T 038e326cc61c4028982856ee90904f00 P 621d11136b6344809ce46c7d56ef6df7 [term 1 FOLLOWER]: Refusing update from remote peer 776c2903f13e4fdfbbd84a818712b42a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:41:39.114564 30616 raft_consensus.cc:1261] T 038e326cc61c4028982856ee90904f00 P dc35b5034cd44955b9f148b99b33214d [term 1 FOLLOWER]: Refusing update from remote peer 776c2903f13e4fdfbbd84a818712b42a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:41:39.114938 30961 consensus_queue.cc:1035] T 038e326cc61c4028982856ee90904f00 P 776c2903f13e4fdfbbd84a818712b42a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc35b5034cd44955b9f148b99b33214d" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 33965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:41:39.115140 30961 consensus_queue.cc:1035] T 038e326cc61c4028982856ee90904f00 P 776c2903f13e4fdfbbd84a818712b42a [LEADER]: Connected to new peer: Peer: permanent_uuid: "621d11136b6344809ce46c7d56ef6df7" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 41785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:41:39.120325 30966 mvcc.cc:205] Tried to move back new op lower bound from 7334206255570386944 to 7334206255416770560. Current Snapshot: MvccSnapshot[applied={T|T < 7334206255570386944}]
I20260928 06:41:39.122099 30967 mvcc.cc:205] Tried to move back new op lower bound from 7334206255570386944 to 7334206255416770560. Current Snapshot: MvccSnapshot[applied={T|T < 7334206255570386944}]
I20260928 06:41:39.122975 30449 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:41:39.123067 30449 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:41:39.123813 30449 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 644d4704ec6844818a6a45b20eeda257: Sending DeleteTablet for 3 replicas of tablet 038e326cc61c4028982856ee90904f00
I20260928 06:41:39.124187 30596 tablet_service.cc:1514] Processing DeleteTablet for tablet 038e326cc61c4028982856ee90904f00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:41:39 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:36716
I20260928 06:41:39.124214 30715 tablet_service.cc:1514] Processing DeleteTablet for tablet 038e326cc61c4028982856ee90904f00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:41:39 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:33652
I20260928 06:41:39.124434 30976 tablet_replica.cc:330] T 038e326cc61c4028982856ee90904f00 P 621d11136b6344809ce46c7d56ef6df7: stopping tablet replica
I20260928 06:41:39.124423 30868 tablet_service.cc:1514] Processing DeleteTablet for tablet 038e326cc61c4028982856ee90904f00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:41:39 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:49926
I20260928 06:41:39.124524 30976 raft_consensus.cc:2229] T 038e326cc61c4028982856ee90904f00 P 621d11136b6344809ce46c7d56ef6df7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:41:39.124631 30976 raft_consensus.cc:2258] T 038e326cc61c4028982856ee90904f00 P 621d11136b6344809ce46c7d56ef6df7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:41:39.125056 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 30531
I20260928 06:41:39.126905 30978 tablet_replica.cc:330] T 038e326cc61c4028982856ee90904f00 P 776c2903f13e4fdfbbd84a818712b42a: stopping tablet replica
I20260928 06:41:39.127056 30978 raft_consensus.cc:2229] T 038e326cc61c4028982856ee90904f00 P 776c2903f13e4fdfbbd84a818712b42a [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:41:39.127238 30978 raft_consensus.cc:2258] T 038e326cc61c4028982856ee90904f00 P 776c2903f13e4fdfbbd84a818712b42a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:41:39.127712 30978 ts_tablet_manager.cc:1905] T 038e326cc61c4028982856ee90904f00 P 776c2903f13e4fdfbbd84a818712b42a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:41:39.127900 30976 ts_tablet_manager.cc:1905] T 038e326cc61c4028982856ee90904f00 P 621d11136b6344809ce46c7d56ef6df7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:41:39.129702 30976 ts_tablet_manager.cc:1918] T 038e326cc61c4028982856ee90904f00 P 621d11136b6344809ce46c7d56ef6df7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:41:39.129817 30976 log.cc:1196] T 038e326cc61c4028982856ee90904f00 P 621d11136b6344809ce46c7d56ef6df7: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-1/wal/wals/038e326cc61c4028982856ee90904f00
I20260928 06:41:39.130326 30976 ts_tablet_manager.cc:1939] T 038e326cc61c4028982856ee90904f00 P 621d11136b6344809ce46c7d56ef6df7: Deleting consensus metadata
I20260928 06:41:39.130774 30437 catalog_manager.cc:4841] TS 621d11136b6344809ce46c7d56ef6df7 (127.18.211.66:41785): tablet 038e326cc61c4028982856ee90904f00 (table test-table [id=d11c315fe75a4828854313f436a69e4f]) successfully deleted
I20260928 06:41:39.130931 30978 ts_tablet_manager.cc:1918] T 038e326cc61c4028982856ee90904f00 P 776c2903f13e4fdfbbd84a818712b42a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:41:39.131042 30978 log.cc:1196] T 038e326cc61c4028982856ee90904f00 P 776c2903f13e4fdfbbd84a818712b42a: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790577602522428-19277-0/minicluster-data/ts-2/wal/wals/038e326cc61c4028982856ee90904f00
I20260928 06:41:39.131387 30978 ts_tablet_manager.cc:1939] T 038e326cc61c4028982856ee90904f00 P 776c2903f13e4fdfbbd84a818712b42a: Deleting consensus metadata
I20260928 06:41:39.131822 30434 catalog_manager.cc:4841] TS 776c2903f13e4fdfbbd84a818712b42a (127.18.211.67:41029): tablet 038e326cc61c4028982856ee90904f00 (table test-table [id=d11c315fe75a4828854313f436a69e4f]) successfully deleted
W20260928 06:41:39.134025 30434 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.211.65:33965 (error 108)
I20260928 06:41:39.134367 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 30668
I20260928 06:41:39.135360 30948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:41:39.135224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:33965 (local address 127.0.0.1:36720)
0928 06:41:39.135275 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 06:41:39.135314 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.65:33965: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 06:41:39.135563 30434 catalog_manager.cc:4584] TS dc35b5034cd44955b9f148b99b33214d (127.18.211.65:33965): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 038e326cc61c4028982856ee90904f00: Network error: Client connection negotiation failed: client connection to 127.18.211.65:33965: connect: Connection refused (error 111)
I20260928 06:41:39.139730 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 30805
I20260928 06:41:39.145567 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 30418
I20260928 06:41:39.150640 19277 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 06:41:39.495013 19277 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 06:41:39.495177 19277 mini_ranger.cc:66] Stopping Ranger...
I20260928 06:41:39.829339 19277 mini_ranger.cc:68] Stopped Ranger
2026-09-28 06:41:39.829 UTC [30004] LOG: received smart shutdown request
2026-09-28 06:41:39.833 UTC [30004] LOG: background worker "logical replication launcher" (PID 30013) exited with exit code 1
2026-09-28 06:41:39.833 UTC [30008] LOG: shutting down
2026-09-28 06:41:39.975 UTC [30004] LOG: database system is shut down
2026-09-28T06:41:39Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46136 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:41:40 dist-test-slave-b28r krb5kdc[30988](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:41:40 dist-test-slave-b28r krb5kdc[30988](info): set up 2 sockets
Sep 28 06:41:40 dist-test-slave-b28r krb5kdc[30988](info): commencing operation
krb5kdc: starting...
W20260928 06:41:41.015926 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:41:41 dist-test-slave-b28r krb5kdc[30988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577701, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:41:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:41:41Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.18.211.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.18.211.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.18.211.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/rangeradmin_127.18.211.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.18.211.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/rangeradmin_127.18.211.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.18.211.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.18.211.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.18.211.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/rangerlookup_127.18.211.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.18.211.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/rangerlookup_127.18.211.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.211.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.211.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.211.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/HTTP_127.18.211.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.211.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/HTTP_127.18.211.84@KRBTEST.COM.keytab.
I20260928 06:41:41.163368 19277 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskOUYCUS/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/postgres -l logfile start
2026-09-28 06:41:43.402 UTC [31018] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 06:41:43.402 UTC [31018] LOG: listening on IPv4 address "127.18.211.84", port 38971
2026-09-28 06:41:43.405 UTC [31018] LOG: listening on Unix socket "/tmp/.s.PGSQL.38971"
2026-09-28 06:41:43.424 UTC [31021] LOG: database system was shut down at 2026-09-28 06:41:42 UTC
2026-09-28 06:41:43.427 UTC [31018] LOG: database system is ready to accept connections
I20260928 06:41:44.299799 19277 mini_postgres.cc:96] Postgres bound to 38971
2026-09-28 06:41:44.305 UTC [31029] FATAL: database "slave" does not exist
127.18.211.84:38971 - accepting connections
I20260928 06:41:44.306020 19277 mini_ranger.cc:161] Starting Ranger...
I20260928 06:41:44.320025 19277 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 06:41:44.718300 19277 mini_ranger.cc:87] Created ranger Postgres database
I20260928 06:41:44.718382 19277 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ranger-admin
2026-09-28 06:41:44,992 [I] DB FLAVOR :POSTGRES
2026-09-28 06:41:44,993 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 06:41:44,993 [I] Checking connection..
2026-09-28 06:41:44,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:38971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 06:41:45,241 [I] Checking connection passed.
2026-09-28 06:41:45,242 [I] --------- Verifying version history table ---------
2026-09-28 06:41:45,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:38971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:41:45,485 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 06:41:45,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:38971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:41:45,738 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 06:41:45,739 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 06:41:45,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:38971/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 06:41:45.979 UTC [31093] WARNING: there is no transaction in progress
2026-09-28 06:41:45,985 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 06:41:45,985 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:38971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 06:41:46,241 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 06:41:46,242 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 06:41:46,242 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:38971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 06:41:46,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:38971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 06:41:46,721 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:38971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-b28r.c.gcp-upstream.internal','N') ;"
2026-09-28 06:41:46,960 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:38971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 06:41:47,197 [I] Table x_portal_user does not exist in database ranger
2026-09-28 06:41:47,198 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:38971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 06:41:47,450 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 06:41:47,450 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:38971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 06:41:47,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:38971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 06:41:47,924 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 06:41:47,924 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:38971/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 06:41:48.840 UTC [31201] WARNING: there is no transaction in progress
2026-09-28 06:41:48.855 UTC [31201] WARNING: there is no transaction in progress
2026-09-28 06:41:48.870 UTC [31201] WARNING: there is no transaction in progress
2026-09-28 06:41:48.885 UTC [31201] WARNING: there is no transaction in progress
2026-09-28 06:41:48.900 UTC [31201] WARNING: there is no transaction in progress
2026-09-28 06:41:48.919 UTC [31201] WARNING: there is no transaction in progress
2026-09-28 06:41:48.973 UTC [31201] WARNING: there is no transaction in progress
2026-09-28 06:41:48.979 UTC [31201] WARNING: there is no transaction in progress
2026-09-28 06:41:48.993 UTC [31201] WARNING: there is no transaction in progress
2026-09-28 06:41:49.003 UTC [31201] WARNING: there is no transaction in progress
2026-09-28 06:41:49.013 UTC [31201] WARNING: there is no transaction in progress
2026-09-28 06:41:49.025 UTC [31201] WARNING: there is no transaction in progress
2026-09-28 06:41:49.034 UTC [31201] WARNING: there is no transaction in progress
2026-09-28 06:41:49.040 UTC [31201] WARNING: there is no transaction in progress
2026-09-28 06:41:49.045 UTC [31201] WARNING: there is no transaction in progress
2026-09-28 06:41:49.051 UTC [31201] WARNING: there is no transaction in progress
2026-09-28 06:41:49,636 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 06:41:49,636 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:38971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 06:41:49,867 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 06:41:49,868 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:38971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 06:41:50,097 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 06:41:50,097 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:38971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 06:41:50,342 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 06:41:50,343 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:38971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 06:41:50,586 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:38971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 06:41:50,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:38971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-b28r.c.gcp-upstream.internal';"
2026-09-28 06:41:51,065 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 06:41:51,066 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:38971/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 06:41:51,291 [I] DB_PATCHES have already been applied
I20260928 06:41:51.297355 19277 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ranger-admin:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskOUYCUS/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 06:41:51.297420 19277 mini_ranger.cc:193] Using host: 127.18.211.84
I20260928 06:41:51.300383 19277 mini_ranger.cc:227] Ranger admin URL: http://127.18.211.84:39999
Sep 28, 2026 6:41:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 6:41:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 6:41:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 6:41:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 6:41:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 6:41:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 6:41:52 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 06:41:52.294513 19277 mini_ranger.cc:160] Time spent starting Ranger: real 7.988s user 0.000s sys 0.008s
Sep 28, 2026 6:42:12 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 6:42:12 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 6:42:12 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 6:42:12 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 6:42:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 6:42:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 6:42:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 6:42:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 6:42:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 6:42:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 6:42:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 6:42:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 6:42:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 6:42:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 6:42:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 6:42:12 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 6:42:12 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 6:42:13 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 06:42:14.376729 19277 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.18.211.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.18.211.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.18.211.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/rangerkms_127.18.211.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.18.211.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/rangerkms_127.18.211.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.211.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.18.211.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.211.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/HTTP_127.18.211.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.211.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/HTTP_127.18.211.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 06:42:14 dist-test-slave-b28r krb5kdc[30988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577734, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 06:42:14.450637 19277 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 06:42:14.458835 19277 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 06:42:15.196295 19277 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 06:42:15.196364 19277 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ranger-kms
I20260928 06:42:15.196378 19277 mini_ranger_kms.cc:227] Using postgres at 127.18.211.84:38971
2026-09-28 06:42:15,411 [I] DB FLAVOR :POSTGRES
2026-09-28 06:42:15,411 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 06:42:15,411 [I] Checking connection
2026-09-28 06:42:15,411 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:38971/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 06:42:15,648 [I] connection success
2026-09-28 06:42:15,649 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 06:42:15,649 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 06:42:15,649 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:38971/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 06:42:15,870 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 06:42:15,871 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 06:42:15,871 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 06:42:15,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.211.84:38971/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 06:42:16,133 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 06:42:16.567165 19277 mini_ranger_kms.cc:310] Created kms service
I20260928 06:42:16.747804 19277 mini_ranger_kms.cc:326] Created kudu user
I20260928 06:42:16.804653 19277 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 06:42:16 dist-test-slave-b28r krb5kdc[30988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577734, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.18.211.84@KRBTEST.COM
I20260928 06:42:17.199717 19277 mini_ranger_kms.cc:384] Added ranger policy
I20260928 06:42:17.199975 19277 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ranger-kms:/tmp/dist-test-taskOUYCUS/build/release/bin/postgresql.jar:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskOUYCUS/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 06:42:17.200006 19277 mini_ranger_kms.cc:240] Using host: 127.18.211.84
I20260928 06:42:17.202965 19277 mini_ranger_kms.cc:276] Ranger KMS PID: 31381
I20260928 06:42:17.203040 19277 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.18.211.84:42675
Sep 28, 2026 6:42:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 6:42:18 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 6:42:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 6:42:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 6:42:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 6:42:18 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 06:42:19.255813 19277 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.805s user 0.000s sys 0.138s
I20260928 06:42:19.255934 19277 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 06:42:19.255991 19277 mini_ranger_kms.cc:401] 127.18.211.84:42675/kms/v1/keys
Sep 28 06:42:19 dist-test-slave-b28r krb5kdc[30988](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790577739, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.211.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 06:42:19 dist-test-slave-b28r krb5kdc[30988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577739, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.211.84@KRBTEST.COM for HTTP/127.18.211.84@KRBTEST.COM
Sep 28 06:42:20 dist-test-slave-b28r krb5kdc[30988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577739, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.211.84@KRBTEST.COM for HTTP/127.18.211.84@KRBTEST.COM
Sep 28 06:42:22 dist-test-slave-b28r krb5kdc[30988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.18.211.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
I20260928 06:42:22.321002 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:35605
--webserver_interface=127.18.211.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:38157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:35605
--ranger_config_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.211.84:42675/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:42:22.400535 31432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:42:22.400743 31432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:42:22.400774 31432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:42:22.402135 31432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:42:22.402195 31432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:42:22.402215 31432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:42:22.402230 31432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:42:22.402254 31432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:42:22.403960 31432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:38157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/master-0/wal
--ranger_kms_url=127.18.211.84:42675/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:35605
--ranger_config_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:35605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:42:22.404210 31432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:42:22.404417 31432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:42:22.409415 31440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:22.409462 31437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:42:22.409533 31432 server_base.cc:1034] running on GCE node
W20260928 06:42:22.409430 31438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:42:22.409898 31432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:42:22.410177 31432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:42:22.411345 31432 hybrid_clock.cc:648] HybridClock initialized: now 1790577742411333 us; error 27 us; skew 500 ppm
Sep 28 06:42:22 dist-test-slave-b28r krb5kdc[30988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577742, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:42:22.413681 31432 init.cc:377] Logged in from keytab as kudu/127.18.211.126@KRBTEST.COM (short username kudu)
I20260928 06:42:22.414108 31432 webserver.cc:460] Webserver started at http://127.18.211.126:41591/ using document root <none> and password file <none>
I20260928 06:42:22.414319 31432 fs_manager.cc:359] Metadata directory not provided
I20260928 06:42:22.414357 31432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:42:22.414475 31432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:42:22 dist-test-slave-b28r krb5kdc[30988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577742, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for HTTP/127.18.211.84@KRBTEST.COM
I20260928 06:42:22.459563 31432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "e1a6ca4f004944f2a90147b9fae23596"
format_stamp: "Formatted at 2026-09-28 06:42:22 on dist-test-slave-b28r"
server_key: "1897b09e0aefa88036d1cc13df3d6f74"
server_key_iv: "f072475261ae86b8d986e03c2fd8fc63"
server_key_version: "kuduclusterkey@0"
I20260928 06:42:22.459949 31432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "e1a6ca4f004944f2a90147b9fae23596"
format_stamp: "Formatted at 2026-09-28 06:42:22 on dist-test-slave-b28r"
server_key: "1897b09e0aefa88036d1cc13df3d6f74"
server_key_iv: "f072475261ae86b8d986e03c2fd8fc63"
server_key_version: "kuduclusterkey@0"
I20260928 06:42:22.461385 31432 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:42:22.476544 31447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:22.476799 31432 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:42:22.476914 31432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "e1a6ca4f004944f2a90147b9fae23596"
format_stamp: "Formatted at 2026-09-28 06:42:22 on dist-test-slave-b28r"
server_key: "1897b09e0aefa88036d1cc13df3d6f74"
server_key_iv: "f072475261ae86b8d986e03c2fd8fc63"
server_key_version: "kuduclusterkey@0"
I20260928 06:42:22.477022 31432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:42:22.497567 31432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:42:22.499603 31432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:42:22.499781 31432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:42:22.504837 31432 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:35605
I20260928 06:42:22.504870 31499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:35605 every 8 connection(s)
I20260928 06:42:22.505262 31432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:42:22.505999 31500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:42:22.506771 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 31432
I20260928 06:42:22.506891 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:42:22.507097 19277 external_mini_cluster.cc:1428] Setting key 32bd9ab420c582aa1cfbe639f517455e
I20260928 06:42:22.509135 31500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596: Bootstrap starting.
I20260928 06:42:22.509841 31500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596: Neither blocks nor log segments found. Creating new log.
I20260928 06:42:22.510131 31500 log.cc:824] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596: Log is configured to *not* fsync() on all Append() calls
I20260928 06:42:22.510859 31500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596: No bootstrap required, opened a new log
I20260928 06:42:22.513049 31500 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1a6ca4f004944f2a90147b9fae23596" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35605 } }
I20260928 06:42:22.513187 31500 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:42:22.513222 31500 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a6ca4f004944f2a90147b9fae23596, State: Initialized, Role: FOLLOWER
I20260928 06:42:22.513384 31500 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1a6ca4f004944f2a90147b9fae23596" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35605 } }
I20260928 06:42:22.513449 31500 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:42:22.513480 31500 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:42:22.513540 31500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:42:22.514046 31500 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1a6ca4f004944f2a90147b9fae23596" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35605 } }
I20260928 06:42:22.514142 31500 leader_election.cc:302] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1a6ca4f004944f2a90147b9fae23596; no voters:
I20260928 06:42:22.514343 31500 leader_election.cc:288] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:42:22.514371 31505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:42:22.514561 31505 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596 [term 1 LEADER]: Becoming Leader. State: Replica: e1a6ca4f004944f2a90147b9fae23596, State: Running, Role: LEADER
Sep 28 06:42:22 dist-test-slave-b28r krb5kdc[30988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:42:22.514689 31500 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:42:22.514684 31505 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1a6ca4f004944f2a90147b9fae23596" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35605 } }
I20260928 06:42:22.515000 31505 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1a6ca4f004944f2a90147b9fae23596" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1a6ca4f004944f2a90147b9fae23596" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35605 } } }
I20260928 06:42:22.515064 31505 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596 [sys.catalog]: This master's current role is: LEADER
I20260928 06:42:22.515213 31505 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1a6ca4f004944f2a90147b9fae23596. Latest consensus state: current_term: 1 leader_uuid: "e1a6ca4f004944f2a90147b9fae23596" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1a6ca4f004944f2a90147b9fae23596" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35605 } } }
I20260928 06:42:22.515260 31505 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596 [sys.catalog]: This master's current role is: LEADER
I20260928 06:42:22.516135 31500 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 06:42:22.520839 31510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:42:22.521059 31503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:22.507662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44694 (local address 127.18.211.126:35605)
0928 06:42:22.507902 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:22.507913 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:22.507947 (+ 34us) server_negotiation.cc:407] Connection header received
0928 06:42:22.508492 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:22.508501 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:22.508564 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:22.509301 (+ 737us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:22.509408 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:22.510644 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:22.511739 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:22.512359 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:22.512472 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:22.515369 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:22.515374 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:22.515374 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:22.515386 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:22.517222 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:22.518738 (+ 1516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:22.518791 (+ 53us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:22.518792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:22.519302 (+ 510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:22.519556 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:22.519557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:22.519557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:22.520028 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:22.520310 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:22.520389 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 06:42:22.520719 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":79,"threads_started":1}
I20260928 06:42:22.522109 31510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:42:22.527542 31510 catalog_manager.cc:1347] Generated new cluster ID: 9a3494e3c6e84cf8ab5f35a9c9e2664c
I20260928 06:42:22.527603 31510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:42:22.549916 31510 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:42:22.550801 31510 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:42:22.559983 31510 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596: Generated new TSK 0
I20260928 06:42:22.560307 31510 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.211.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab.
Entry for principal kudu/127.18.211.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab.
I20260928 06:42:23.213438 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.211.84:42675/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.65:0
--local_ip_for_outbound_sockets=127.18.211.65
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35605
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:38157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:42:23.304025 31545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:42:23.304243 31545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:42:23.304277 31545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:42:23.305485 31545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:42:23.305533 31545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:42:23.305569 31545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.65
I20260928 06:42:23.307130 31545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:38157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.18.211.84:42675/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.65
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:42:23.307371 31545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:42:23.307588 31545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:42:23.312532 31553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:23.312645 31551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:23.312549 31550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:42:23.313171 31545 server_base.cc:1034] running on GCE node
I20260928 06:42:23.313479 31545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:42:23.313727 31545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:42:23.314915 31545 hybrid_clock.cc:648] HybridClock initialized: now 1790577743314899 us; error 27 us; skew 500 ppm
Sep 28 06:42:23 dist-test-slave-b28r krb5kdc[30988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577743, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:42:23.317701 31545 init.cc:377] Logged in from keytab as kudu/127.18.211.65@KRBTEST.COM (short username kudu)
I20260928 06:42:23.318112 31545 webserver.cc:460] Webserver started at http://127.18.211.65:38055/ using document root <none> and password file <none>
I20260928 06:42:23.318382 31545 fs_manager.cc:359] Metadata directory not provided
I20260928 06:42:23.318429 31545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:42:23.318576 31545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:42:23 dist-test-slave-b28r krb5kdc[30988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577743, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for HTTP/127.18.211.84@KRBTEST.COM
I20260928 06:42:23.332644 31545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-0/data/instance:
uuid: "818b03aed0a74de3a6055273ce184e26"
format_stamp: "Formatted at 2026-09-28 06:42:23 on dist-test-slave-b28r"
server_key: "a5581e1aeeeba10624080a241934e9aa"
server_key_iv: "7b025dbdde13efcc640d99763644d9f9"
server_key_version: "kuduclusterkey@0"
I20260928 06:42:23.333396 31545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance:
uuid: "818b03aed0a74de3a6055273ce184e26"
format_stamp: "Formatted at 2026-09-28 06:42:23 on dist-test-slave-b28r"
server_key: "a5581e1aeeeba10624080a241934e9aa"
server_key_iv: "7b025dbdde13efcc640d99763644d9f9"
server_key_version: "kuduclusterkey@0"
I20260928 06:42:23.335237 31545 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 06:42:23.355244 31560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:23.355552 31545 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260928 06:42:23.355697 31545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-0/wal
uuid: "818b03aed0a74de3a6055273ce184e26"
format_stamp: "Formatted at 2026-09-28 06:42:23 on dist-test-slave-b28r"
server_key: "a5581e1aeeeba10624080a241934e9aa"
server_key_iv: "7b025dbdde13efcc640d99763644d9f9"
server_key_version: "kuduclusterkey@0"
I20260928 06:42:23.355880 31545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:42:23.371492 31545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:42:23.373874 31545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:42:23.374037 31545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:42:23.374294 31545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:42:23.374720 31545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:42:23.374783 31545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:23.374819 31545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:42:23.374897 31545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:23.382880 31545 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.65:36373
I20260928 06:42:23.383383 31545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
I20260928 06:42:23.384364 31673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.65:36373 every 8 connection(s)
I20260928 06:42:23.391759 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 31545
I20260928 06:42:23.391866 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance
I20260928 06:42:23.392048 19277 external_mini_cluster.cc:1428] Setting key 8f723430c4c18b2c0e22200e331ec380
Sep 28 06:42:23 dist-test-slave-b28r krb5kdc[30988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577743, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:42:23.396548 31676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:23.384776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35605 (local address 127.18.211.65:40613)
0928 06:42:23.385115 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:23.385160 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:23.385729 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:23.386102 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:23.386105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:23.386297 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:42:23.386495 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:23.386499 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:23.387297 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:23.387298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:23.389511 (+ 2213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:23.389515 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:23.390155 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:23.390156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:23.390297 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:23.390928 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:42:23.390939 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:42:23.392672 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:42:23.394827 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:23.394834 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:23.394837 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:23.395175 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:23.395423 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:23.395424 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:23.395424 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:23.395551 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:23.395869 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:42:23.395876 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:42:23.396039 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 06:42:23.396153 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 06:42:23.396305 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":87,"threads_started":1}
I20260928 06:42:23.401136 31677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:23.384803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:40613 (local address 127.18.211.126:35605)
0928 06:42:23.385896 (+ 1093us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:23.385898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:23.385906 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:42:23.385928 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:23.385929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:23.385940 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:23.386008 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:23.386600 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:23.387174 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:23.389653 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:23.389983 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:23.390051 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:23.392783 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:23.392785 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:23.392785 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:23.392793 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:23.394721 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:23.395273 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:23.395275 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:23.395275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:23.395347 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:23.395622 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:23.395633 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:23.395634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:23.395776 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:23.395840 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:23.400932 (+ 5092us) server_negotiation.cc:299] Negotiation successful
0928 06:42:23.401029 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1061,"thread_start_us":83,"threads_started":1}
I20260928 06:42:23.401726 31674 heartbeater.cc:347] Connected to a master server at 127.18.211.126:35605
I20260928 06:42:23.401829 31674 heartbeater.cc:464] Registering TS with master...
I20260928 06:42:23.402007 31674 heartbeater.cc:511] Master 127.18.211.126:35605 requested a full tablet report, sending...
I20260928 06:42:23.402565 31464 ts_manager.cc:194] Registered new tserver with Master: 818b03aed0a74de3a6055273ce184e26 (127.18.211.65:36373)
WARNING: no policy specified for kudu/127.18.211.66@KRBTEST.COM; defaulting to no policy
I20260928 06:42:23.403647 31464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.65@KRBTEST.COM'} at 127.18.211.65:40613
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab.
Entry for principal kudu/127.18.211.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab.
I20260928 06:42:23.419950 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.211.84:42675/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.66:0
--local_ip_for_outbound_sockets=127.18.211.66
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35605
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:38157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:42:23.534284 31682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:42:23.534551 31682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:42:23.534584 31682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:42:23.536242 31682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:42:23.536302 31682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:42:23.536339 31682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.66
I20260928 06:42:23.538470 31682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:38157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.18.211.84:42675/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.66
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:42:23.538866 31682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:42:23.539175 31682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:42:23.545213 31689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:23.545297 31690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:23.545204 31692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:42:23.546258 31682 server_base.cc:1034] running on GCE node
I20260928 06:42:23.546487 31682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:42:23.546751 31682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:42:23.547924 31682 hybrid_clock.cc:648] HybridClock initialized: now 1790577743547892 us; error 52 us; skew 500 ppm
Sep 28 06:42:23 dist-test-slave-b28r krb5kdc[30988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577743, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:42:23.550984 31682 init.cc:377] Logged in from keytab as kudu/127.18.211.66@KRBTEST.COM (short username kudu)
I20260928 06:42:23.551509 31682 webserver.cc:460] Webserver started at http://127.18.211.66:41251/ using document root <none> and password file <none>
I20260928 06:42:23.551769 31682 fs_manager.cc:359] Metadata directory not provided
I20260928 06:42:23.551824 31682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:42:23.551991 31682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:42:23 dist-test-slave-b28r krb5kdc[30988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577743, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for HTTP/127.18.211.84@KRBTEST.COM
I20260928 06:42:23.565557 31682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-1/data/instance:
uuid: "6adf7c75f1fe44cb825f8761b44ec495"
format_stamp: "Formatted at 2026-09-28 06:42:23 on dist-test-slave-b28r"
server_key: "06a5b4476e45097d723c7bc57ac5698b"
server_key_iv: "9fb6458b292d9371f1d332f34c946de5"
server_key_version: "kuduclusterkey@0"
I20260928 06:42:23.566013 31682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance:
uuid: "6adf7c75f1fe44cb825f8761b44ec495"
format_stamp: "Formatted at 2026-09-28 06:42:23 on dist-test-slave-b28r"
server_key: "06a5b4476e45097d723c7bc57ac5698b"
server_key_iv: "9fb6458b292d9371f1d332f34c946de5"
server_key_version: "kuduclusterkey@0"
I20260928 06:42:23.567766 31682 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 06:42:23.583207 31699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:23.583461 31682 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260928 06:42:23.583567 31682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-1/wal
uuid: "6adf7c75f1fe44cb825f8761b44ec495"
format_stamp: "Formatted at 2026-09-28 06:42:23 on dist-test-slave-b28r"
server_key: "06a5b4476e45097d723c7bc57ac5698b"
server_key_iv: "9fb6458b292d9371f1d332f34c946de5"
server_key_version: "kuduclusterkey@0"
I20260928 06:42:23.583663 31682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 28 06:42:23 dist-test-slave-b28r krb5kdc[30988](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790577743, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:42:23.600373 31682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:42:23.602680 31682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:42:23.602907 31682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:42:23.603286 31682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:42:23.603708 31682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:42:23.603816 31682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:23.603892 31682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:42:23.603946 31682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:23.610662 31682 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.66:35049
I20260928 06:42:23.610759 31812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.66:35049 every 8 connection(s)
I20260928 06:42:23.611181 31682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
I20260928 06:42:23.617393 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 31682
I20260928 06:42:23.617507 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance
I20260928 06:42:23.617707 19277 external_mini_cluster.cc:1428] Setting key 2c8f9e6d446f2357581651ef50ef43a1
Sep 28 06:42:23 dist-test-slave-b28r krb5kdc[30988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577743, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:42:23.622455 31677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:23.611971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:52455 (local address 127.18.211.126:35605)
0928 06:42:23.612017 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:23.612019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:23.612257 (+ 238us) server_negotiation.cc:407] Connection header received
0928 06:42:23.612919 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:23.612920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:23.612936 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:23.613002 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:23.613696 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:23.614476 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:23.616005 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:23.616313 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:23.616357 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:23.619203 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:23.619205 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:23.619205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:23.619214 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:23.620895 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:23.621366 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:23.621368 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:23.621368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:23.621432 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:23.621702 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:23.621703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:23.621703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:23.621849 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:23.621891 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:23.622260 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 06:42:23.622355 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:42:23.623220 31815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:23.611948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35605 (local address 127.18.211.66:52455)
0928 06:42:23.612112 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:23.612161 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:23.612807 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:23.613078 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:23.613083 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:23.613332 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:42:23.613605 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:23.613611 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:23.614587 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:23.614588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:23.615902 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:23.615905 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:23.616466 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:23.616467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:23.616606 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:23.617177 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:42:23.617187 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:42:23.619060 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:42:23.620985 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:23.620992 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:23.620995 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:23.621277 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:23.621499 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:23.621500 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:23.621500 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:23.621632 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:23.621935 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:42:23.621942 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:42:23.622105 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 06:42:23.622778 (+ 673us) client_negotiation.cc:240] Negotiation successful
0928 06:42:23.622954 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
I20260928 06:42:23.623888 31813 heartbeater.cc:347] Connected to a master server at 127.18.211.126:35605
I20260928 06:42:23.623986 31813 heartbeater.cc:464] Registering TS with master...
I20260928 06:42:23.624208 31813 heartbeater.cc:511] Master 127.18.211.126:35605 requested a full tablet report, sending...
I20260928 06:42:23.624627 31464 ts_manager.cc:194] Registered new tserver with Master: 6adf7c75f1fe44cb825f8761b44ec495 (127.18.211.66:35049)
I20260928 06:42:23.625131 31464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.66@KRBTEST.COM'} at 127.18.211.66:52455
WARNING: no policy specified for kudu/127.18.211.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab.
Entry for principal kudu/127.18.211.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab.
I20260928 06:42:23.653887 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.211.84:42675/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.67:0
--local_ip_for_outbound_sockets=127.18.211.67
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35605
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:38157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:42:23.776206 31818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:42:23.776481 31818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:42:23.776515 31818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:42:23.778199 31818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:42:23.778264 31818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:42:23.778303 31818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.67
I20260928 06:42:23.780678 31818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:38157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.18.211.84:42675/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.67
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:42:23.780961 31818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:42:23.781222 31818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:42:23.788702 31827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:42:23.789013 31818 server_base.cc:1034] running on GCE node
W20260928 06:42:23.789402 31824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:23.790306 31825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:42:23.790629 31818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:42:23.790918 31818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:42:23.792068 31818 hybrid_clock.cc:648] HybridClock initialized: now 1790577743792046 us; error 36 us; skew 500 ppm
Sep 28 06:42:23 dist-test-slave-b28r krb5kdc[30988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577743, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:42:23.795387 31818 init.cc:377] Logged in from keytab as kudu/127.18.211.67@KRBTEST.COM (short username kudu)
I20260928 06:42:23.795895 31818 webserver.cc:460] Webserver started at http://127.18.211.67:44461/ using document root <none> and password file <none>
I20260928 06:42:23.796150 31818 fs_manager.cc:359] Metadata directory not provided
I20260928 06:42:23.796203 31818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:42:23.796332 31818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 06:42:23 dist-test-slave-b28r krb5kdc[30988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577743, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for HTTP/127.18.211.84@KRBTEST.COM
I20260928 06:42:23.813253 31818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-2/data/instance:
uuid: "c6a47d96a5c14791bfdc2e0f58a759c8"
format_stamp: "Formatted at 2026-09-28 06:42:23 on dist-test-slave-b28r"
server_key: "64170f994b19c4f2f633cf8dadc5e7e6"
server_key_iv: "ed2a47c35e5af354c75b4b73d611ad24"
server_key_version: "kuduclusterkey@0"
I20260928 06:42:23.813709 31818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance:
uuid: "c6a47d96a5c14791bfdc2e0f58a759c8"
format_stamp: "Formatted at 2026-09-28 06:42:23 on dist-test-slave-b28r"
server_key: "64170f994b19c4f2f633cf8dadc5e7e6"
server_key_iv: "ed2a47c35e5af354c75b4b73d611ad24"
server_key_version: "kuduclusterkey@0"
I20260928 06:42:23.815953 31818 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260928 06:42:23.835264 31834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:23.835530 31818 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260928 06:42:23.835662 31818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-2/wal
uuid: "c6a47d96a5c14791bfdc2e0f58a759c8"
format_stamp: "Formatted at 2026-09-28 06:42:23 on dist-test-slave-b28r"
server_key: "64170f994b19c4f2f633cf8dadc5e7e6"
server_key_iv: "ed2a47c35e5af354c75b4b73d611ad24"
server_key_version: "kuduclusterkey@0"
I20260928 06:42:23.835824 31818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:42:23.854707 31818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:42:23.862859 31818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:42:23.863098 31818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:42:23.863416 31818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:42:23.863849 31818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:42:23.863955 31818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:23.864024 31818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:42:23.864081 31818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:23.874184 31818 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.67:44515
I20260928 06:42:23.874734 31818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
I20260928 06:42:23.875804 31947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.67:44515 every 8 connection(s)
I20260928 06:42:23.881456 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 31818
I20260928 06:42:23.881556 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance
I20260928 06:42:23.881742 19277 external_mini_cluster.cc:1428] Setting key 4e3d25b36133eed8dc19e5a787efcdcc
Sep 28 06:42:23 dist-test-slave-b28r krb5kdc[30988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577743, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:42:23.885747 31677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:23.877195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:53965 (local address 127.18.211.126:35605)
0928 06:42:23.877246 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:23.877248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:23.877552 (+ 304us) server_negotiation.cc:407] Connection header received
0928 06:42:23.878046 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:23.878047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:23.878060 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:23.878120 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:23.878696 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:23.879352 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:23.880318 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:23.880520 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:23.880575 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:23.882873 (+ 2298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:23.882874 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:23.882874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:23.882881 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:23.884241 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:23.884668 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:23.884669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:23.884670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:23.884733 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:23.885037 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:23.885038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:23.885039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:23.885175 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:23.885216 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:23.885572 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 06:42:23.885656 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:42:23.886468 31950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:23.877131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35605 (local address 127.18.211.67:53965)
0928 06:42:23.877422 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:23.877458 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:23.877952 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:23.878191 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:23.878195 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:23.878416 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:42:23.878611 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:23.878615 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:23.879448 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:23.879448 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:23.880231 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:23.880233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:23.880605 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:23.880606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:23.880758 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:23.881230 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:42:23.881239 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:42:23.882794 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:42:23.884323 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:23.884330 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:23.884332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:23.884577 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:23.884824 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:23.884824 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:23.884825 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:23.884959 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:23.885259 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:42:23.885265 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:42:23.885418 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 06:42:23.886175 (+ 757us) client_negotiation.cc:240] Negotiation successful
0928 06:42:23.886306 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":114,"threads_started":1}
I20260928 06:42:23.886893 31948 heartbeater.cc:347] Connected to a master server at 127.18.211.126:35605
I20260928 06:42:23.886987 31948 heartbeater.cc:464] Registering TS with master...
I20260928 06:42:23.887219 31948 heartbeater.cc:511] Master 127.18.211.126:35605 requested a full tablet report, sending...
I20260928 06:42:23.887570 31464 ts_manager.cc:194] Registered new tserver with Master: c6a47d96a5c14791bfdc2e0f58a759c8 (127.18.211.67:44515)
I20260928 06:42:23.888003 31464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.67@KRBTEST.COM'} at 127.18.211.67:53965
I20260928 06:42:23.896865 19277 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:42:23.897398 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 31545
I20260928 06:42:23.902102 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 31682
I20260928 06:42:23.907946 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 31818
I20260928 06:42:23.912151 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 31432
I20260928 06:42:23.916453 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:35605
--webserver_interface=127.18.211.126
--webserver_port=41591
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:38157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:35605
--ranger_config_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.211.84:42675/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.211.84:42675/kms with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:42:23.995054 31951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:42:23.995285 31951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:42:23.995313 31951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:42:23.996447 31951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:42:23.996488 31951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:42:23.996505 31951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:42:23.996520 31951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:42:23.996534 31951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:42:23.998064 31951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:38157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.211.84:42675/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:35605
--ranger_config_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:35605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=41591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:42:23.998304 31951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:42:23.998497 31951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:42:24.003029 31959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:24.003047 31957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:24.003041 31956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:42:24.003279 31951 server_base.cc:1034] running on GCE node
I20260928 06:42:24.003504 31951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:42:24.003774 31951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:42:24.004925 31951 hybrid_clock.cc:648] HybridClock initialized: now 1790577744004907 us; error 28 us; skew 500 ppm
Sep 28 06:42:24 dist-test-slave-b28r krb5kdc[30988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577744, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:42:24.007265 31951 init.cc:377] Logged in from keytab as kudu/127.18.211.126@KRBTEST.COM (short username kudu)
I20260928 06:42:24.007640 31951 webserver.cc:460] Webserver started at http://127.18.211.126:41591/ using document root <none> and password file <none>
I20260928 06:42:24.007851 31951 fs_manager.cc:359] Metadata directory not provided
I20260928 06:42:24.007911 31951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 06:42:24 dist-test-slave-b28r krb5kdc[30988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577744, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for HTTP/127.18.211.84@KRBTEST.COM
I20260928 06:42:24.061444 31951 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:42:24.062168 31967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:24.062318 31951 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:42:24.062405 31951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "e1a6ca4f004944f2a90147b9fae23596"
format_stamp: "Formatted at 2026-09-28 06:42:22 on dist-test-slave-b28r"
server_key: "1897b09e0aefa88036d1cc13df3d6f74"
server_key_iv: "f072475261ae86b8d986e03c2fd8fc63"
server_key_version: "kuduclusterkey@0"
I20260928 06:42:24.062827 31951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:42:24.086825 31951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:42:24.088809 31951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:42:24.088954 31951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:42:24.093034 31951 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:35605
I20260928 06:42:24.093087 32019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:35605 every 8 connection(s)
I20260928 06:42:24.093467 31951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:42:24.094662 32020 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:42:24.095353 32020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596: Bootstrap starting.
I20260928 06:42:24.102113 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 31951
I20260928 06:42:24.102495 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.211.84:42675/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.65:36373
--local_ip_for_outbound_sockets=127.18.211.65
--tserver_master_addrs=127.18.211.126:35605
--webserver_port=38055
--webserver_interface=127.18.211.65
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:38157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.211.84:42675/kms with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:42:24.103302 32020 log.cc:824] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596: Log is configured to *not* fsync() on all Append() calls
I20260928 06:42:24.105124 32020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:42:24.105371 32020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596: Bootstrap complete.
I20260928 06:42:24.107705 32020 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1a6ca4f004944f2a90147b9fae23596" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35605 } }
I20260928 06:42:24.107950 32020 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a6ca4f004944f2a90147b9fae23596, State: Initialized, Role: FOLLOWER
I20260928 06:42:24.108111 32020 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1a6ca4f004944f2a90147b9fae23596" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35605 } }
I20260928 06:42:24.108186 32020 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:42:24.108233 32020 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:42:24.108280 32020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596 [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:42:24.109246 32020 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1a6ca4f004944f2a90147b9fae23596" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35605 } }
I20260928 06:42:24.109345 32020 leader_election.cc:302] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1a6ca4f004944f2a90147b9fae23596; no voters:
I20260928 06:42:24.109499 32020 leader_election.cc:288] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 06:42:24.109544 32024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:42:24.109673 32024 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596 [term 2 LEADER]: Becoming Leader. State: Replica: e1a6ca4f004944f2a90147b9fae23596, State: Running, Role: LEADER
I20260928 06:42:24.109771 32020 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:42:24.109822 32024 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1a6ca4f004944f2a90147b9fae23596" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35605 } }
I20260928 06:42:24.110121 32025 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e1a6ca4f004944f2a90147b9fae23596" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1a6ca4f004944f2a90147b9fae23596" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35605 } } }
I20260928 06:42:24.110217 32025 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596 [sys.catalog]: This master's current role is: LEADER
I20260928 06:42:24.110359 32024 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1a6ca4f004944f2a90147b9fae23596. Latest consensus state: current_term: 2 leader_uuid: "e1a6ca4f004944f2a90147b9fae23596" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1a6ca4f004944f2a90147b9fae23596" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35605 } } }
I20260928 06:42:24.110409 32024 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596 [sys.catalog]: This master's current role is: LEADER
I20260928 06:42:24.110966 32030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:42:24.111194 32020 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 06:42:24.111500 32030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:42:24.113667 32030 catalog_manager.cc:1259] Loaded cluster ID: 9a3494e3c6e84cf8ab5f35a9c9e2664c
I20260928 06:42:24.113826 32030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:42:24.115691 32030 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:42:24.115955 32030 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596: Loaded TSK: 0
I20260928 06:42:24.116325 32030 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:42:24.189373 32023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:42:24.189707 32023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:42:24.189812 32023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:42:24.191219 32023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:42:24.191267 32023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:42:24.191305 32023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.65
I20260928 06:42:24.192828 32023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:38157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.211.84:42675/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.65:36373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.211.65
--webserver_port=38055
--tserver_master_addrs=127.18.211.126:35605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.65
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:42:24.193073 32023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:42:24.193331 32023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:42:24.198915 32060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:24.199254 32057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:24.199397 32058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:42:24.200102 32023 server_base.cc:1034] running on GCE node
I20260928 06:42:24.200325 32023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:42:24.200564 32023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:42:24.201722 32023 hybrid_clock.cc:648] HybridClock initialized: now 1790577744201707 us; error 28 us; skew 500 ppm
Sep 28 06:42:24 dist-test-slave-b28r krb5kdc[30988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577744, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:42:24.204452 32023 init.cc:377] Logged in from keytab as kudu/127.18.211.65@KRBTEST.COM (short username kudu)
I20260928 06:42:24.204929 32023 webserver.cc:460] Webserver started at http://127.18.211.65:38055/ using document root <none> and password file <none>
I20260928 06:42:24.205215 32023 fs_manager.cc:359] Metadata directory not provided
I20260928 06:42:24.205298 32023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 06:42:24 dist-test-slave-b28r krb5kdc[30988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577744, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for HTTP/127.18.211.84@KRBTEST.COM
I20260928 06:42:24.223014 32023 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260928 06:42:24.223819 32068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:24.223994 32023 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:42:24.224088 32023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-0/wal
uuid: "818b03aed0a74de3a6055273ce184e26"
format_stamp: "Formatted at 2026-09-28 06:42:23 on dist-test-slave-b28r"
server_key: "a5581e1aeeeba10624080a241934e9aa"
server_key_iv: "7b025dbdde13efcc640d99763644d9f9"
server_key_version: "kuduclusterkey@0"
I20260928 06:42:24.224475 32023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:42:24.237437 32023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:42:24.239476 32023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:42:24.239630 32023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:42:24.239873 32023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:42:24.240235 32023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:42:24.240275 32023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:24.240305 32023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:42:24.240342 32023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:24.247067 32023 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.65:36373
I20260928 06:42:24.247327 32181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.65:36373 every 8 connection(s)
I20260928 06:42:24.247691 32023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
I20260928 06:42:24.248303 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 32023
I20260928 06:42:24.248557 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.211.84:42675/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.66:35049
--local_ip_for_outbound_sockets=127.18.211.66
--tserver_master_addrs=127.18.211.126:35605
--webserver_port=41251
--webserver_interface=127.18.211.66
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:38157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.211.84:42675/kms with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:42:24 dist-test-slave-b28r krb5kdc[30988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577744, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:42:24.263654 32185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:24.250313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35605 (local address 127.18.211.65:42955)
0928 06:42:24.251184 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:24.251223 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:24.251661 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:24.253265 (+ 1604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:24.253269 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:24.253516 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:42:24.253813 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:24.253820 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:24.254964 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:24.254965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:24.256055 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:24.256059 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:24.257028 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:24.257029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:24.257183 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:24.257793 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:42:24.257803 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:42:24.259454 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:42:24.261662 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:24.261676 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:24.261678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:24.261975 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:24.262275 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:24.262276 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:24.262276 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:24.262425 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:24.262948 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:42:24.262955 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:42:24.263144 (+ 189us) client_negotiation.cc:769] Sending connection context
0928 06:42:24.263248 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 06:42:24.263413 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"thread_start_us":88,"threads_started":1}
I20260928 06:42:24.264343 32186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:24.250697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:42955 (local address 127.18.211.126:35605)
0928 06:42:24.252716 (+ 2019us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:24.252725 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:24.252757 (+ 32us) server_negotiation.cc:407] Connection header received
0928 06:42:24.252884 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:24.252890 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:24.252959 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:24.253093 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:24.253950 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:24.254865 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:24.256160 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:24.256803 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:24.256914 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:24.259605 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:24.259610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:24.259610 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:24.259625 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:24.261567 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:24.262079 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:24.262082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:24.262082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:24.262204 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:24.262507 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:24.262507 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:24.262508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:24.262875 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:24.262967 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:24.263980 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0928 06:42:24.264104 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1957,"thread_start_us":84,"threads_started":1}
I20260928 06:42:24.265045 32182 heartbeater.cc:347] Connected to a master server at 127.18.211.126:35605
I20260928 06:42:24.265180 32182 heartbeater.cc:464] Registering TS with master...
I20260928 06:42:24.265391 32182 heartbeater.cc:511] Master 127.18.211.126:35605 requested a full tablet report, sending...
W20260928 06:42:24.266347 32182 heartbeater.cc:644] Failed to heartbeat to 127.18.211.126:35605 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 06:42:24.266456 32182 heartbeater.cc:464] Registering TS with master...
I20260928 06:42:24.266546 32182 heartbeater.cc:511] Master 127.18.211.126:35605 requested a full tablet report, sending...
I20260928 06:42:24.266932 32182 heartbeater.cc:464] Registering TS with master...
I20260928 06:42:24.267035 32182 heartbeater.cc:511] Master 127.18.211.126:35605 requested a full tablet report, sending...
W20260928 06:42:24.267402 32182 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:42:24.360056 32184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:42:24.360318 32184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:42:24.360352 32184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:42:24.362033 32184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:42:24.362128 32184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:42:24.362183 32184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.66
I20260928 06:42:24.364326 32184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:38157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.211.84:42675/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.66:35049
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.211.66
--webserver_port=41251
--tserver_master_addrs=127.18.211.126:35605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.66
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:42:24.364699 32184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:42:24.364998 32184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:42:24.371794 32191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:24.371794 32192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:24.371990 32194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:42:24.372604 32184 server_base.cc:1034] running on GCE node
I20260928 06:42:24.372788 32184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:42:24.373021 32184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:42:24.374166 32184 hybrid_clock.cc:648] HybridClock initialized: now 1790577744374148 us; error 32 us; skew 500 ppm
Sep 28 06:42:24 dist-test-slave-b28r krb5kdc[30988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577744, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:42:24.380466 32184 init.cc:377] Logged in from keytab as kudu/127.18.211.66@KRBTEST.COM (short username kudu)
I20260928 06:42:24.381065 32184 webserver.cc:460] Webserver started at http://127.18.211.66:41251/ using document root <none> and password file <none>
I20260928 06:42:24.381350 32184 fs_manager.cc:359] Metadata directory not provided
I20260928 06:42:24.381439 32184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 06:42:24 dist-test-slave-b28r krb5kdc[30988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577744, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for HTTP/127.18.211.84@KRBTEST.COM
I20260928 06:42:24.397347 32184 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260928 06:42:24.398399 32202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:24.398561 32184 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:42:24.398640 32184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-1/wal
uuid: "6adf7c75f1fe44cb825f8761b44ec495"
format_stamp: "Formatted at 2026-09-28 06:42:23 on dist-test-slave-b28r"
server_key: "06a5b4476e45097d723c7bc57ac5698b"
server_key_iv: "9fb6458b292d9371f1d332f34c946de5"
server_key_version: "kuduclusterkey@0"
I20260928 06:42:24.399116 32184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:42:24.414039 32184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:42:24.416308 32184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:42:24.416512 32184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:42:24.416728 32184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:42:24.417080 32184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:42:24.417117 32184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:24.417147 32184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:42:24.417184 32184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:24.423707 32184 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.66:35049
I20260928 06:42:24.424080 32184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
I20260928 06:42:24.424635 32315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.66:35049 every 8 connection(s)
Sep 28 06:42:24 dist-test-slave-b28r krb5kdc[30988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577744, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:42:24.432765 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 32184
I20260928 06:42:24.433061 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.211.84:42675/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.67:44515
--local_ip_for_outbound_sockets=127.18.211.67
--tserver_master_addrs=127.18.211.126:35605
--webserver_port=44461
--webserver_interface=127.18.211.67
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:38157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.211.84:42675/kms with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:42:24.433336 32186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:24.424843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:51933 (local address 127.18.211.126:35605)
0928 06:42:24.424890 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:24.424892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:24.425255 (+ 363us) server_negotiation.cc:407] Connection header received
0928 06:42:24.425739 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:24.425740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:24.425754 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:24.425806 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:24.426327 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:24.426821 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:24.428107 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:24.428314 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:24.428374 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:24.430513 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:24.430514 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:24.430515 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:24.430523 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:24.431881 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:24.432372 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:24.432374 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:24.432374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:24.432437 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:24.432663 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:24.432664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:24.432664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:24.432778 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:24.432831 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:24.433157 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 06:42:24.433251 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:42:24.433764 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:24.424975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35605 (local address 127.18.211.66:51933)
0928 06:42:24.425127 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:24.425169 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:24.425641 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:24.425886 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:24.425890 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:24.426054 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:42:24.426237 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:24.426241 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:24.426909 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:24.426910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:24.428021 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:24.428024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:24.428416 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:24.428418 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:24.428523 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:24.428998 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:42:24.429008 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:42:24.430427 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:42:24.431957 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:24.431966 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:24.431968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:24.432265 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:24.432493 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:24.432494 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:24.432494 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:24.432607 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:24.432900 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:42:24.432908 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:42:24.433030 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 06:42:24.433326 (+ 296us) client_negotiation.cc:240] Negotiation successful
0928 06:42:24.433598 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
I20260928 06:42:24.434193 32316 heartbeater.cc:347] Connected to a master server at 127.18.211.126:35605
I20260928 06:42:24.434288 32316 heartbeater.cc:464] Registering TS with master...
I20260928 06:42:24.434463 32316 heartbeater.cc:511] Master 127.18.211.126:35605 requested a full tablet report, sending...
W20260928 06:42:24.435067 32316 heartbeater.cc:644] Failed to heartbeat to 127.18.211.126:35605 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 06:42:24.435218 32316 heartbeater.cc:464] Registering TS with master...
I20260928 06:42:24.435328 32316 heartbeater.cc:511] Master 127.18.211.126:35605 requested a full tablet report, sending...
I20260928 06:42:24.435657 32316 heartbeater.cc:464] Registering TS with master...
I20260928 06:42:24.435760 32316 heartbeater.cc:511] Master 127.18.211.126:35605 requested a full tablet report, sending...
W20260928 06:42:24.436062 32316 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:42:24.545714 32319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:42:24.546003 32319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:42:24.546041 32319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:42:24.547777 32319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:42:24.547840 32319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:42:24.547894 32319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.67
I20260928 06:42:24.550100 32319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:38157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.211.84:42675/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.67:44515
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.211.67
--webserver_port=44461
--tserver_master_addrs=127.18.211.126:35605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.67
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:42:24.550490 32319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:42:24.550773 32319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:42:24.559798 32327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:24.559799 32325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:24.560796 32324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:42:24.561103 32319 server_base.cc:1034] running on GCE node
I20260928 06:42:24.561328 32319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:42:24.561622 32319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:42:24.562801 32319 hybrid_clock.cc:648] HybridClock initialized: now 1790577744562780 us; error 32 us; skew 500 ppm
Sep 28 06:42:24 dist-test-slave-b28r krb5kdc[30988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577744, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:42:24.566185 32319 init.cc:377] Logged in from keytab as kudu/127.18.211.67@KRBTEST.COM (short username kudu)
I20260928 06:42:24.566803 32319 webserver.cc:460] Webserver started at http://127.18.211.67:44461/ using document root <none> and password file <none>
I20260928 06:42:24.567102 32319 fs_manager.cc:359] Metadata directory not provided
I20260928 06:42:24.567189 32319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 06:42:24 dist-test-slave-b28r krb5kdc[30988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577744, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for HTTP/127.18.211.84@KRBTEST.COM
I20260928 06:42:24.583971 32319 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:42:24.585139 32335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:24.585367 32319 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:42:24.585505 32319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-2/wal
uuid: "c6a47d96a5c14791bfdc2e0f58a759c8"
format_stamp: "Formatted at 2026-09-28 06:42:23 on dist-test-slave-b28r"
server_key: "64170f994b19c4f2f633cf8dadc5e7e6"
server_key_iv: "ed2a47c35e5af354c75b4b73d611ad24"
server_key_version: "kuduclusterkey@0"
I20260928 06:42:24.586016 32319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:42:24.604190 32319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:42:24.606700 32319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:42:24.606904 32319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:42:24.607228 32319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:42:24.607656 32319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:42:24.607743 32319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:24.607810 32319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:42:24.607856 32319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:24.620646 32319 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.67:44515
I20260928 06:42:24.620711 32451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.67:44515 every 8 connection(s)
I20260928 06:42:24.621152 32319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
I20260928 06:42:24.621416 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 32319
Sep 28 06:42:24 dist-test-slave-b28r krb5kdc[30988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577744, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:42:24.633414 32455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:24.621896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:35825 (local address 127.18.211.126:35605)
0928 06:42:24.622024 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:24.622026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:24.622237 (+ 211us) server_negotiation.cc:407] Connection header received
0928 06:42:24.622888 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:24.622890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:24.622906 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:24.622977 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:24.623728 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:24.624223 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:24.625710 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:24.625914 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:24.625971 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:24.628550 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:24.628552 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:24.628552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:24.628560 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:24.630132 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:24.632076 (+ 1944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:24.632077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:24.632077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:24.632141 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:24.632560 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:24.632561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:24.632562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:24.632730 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:24.632815 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:24.633230 (+ 415us) server_negotiation.cc:299] Negotiation successful
0928 06:42:24.633303 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":116,"thread_start_us":58,"threads_started":1}
I20260928 06:42:24.633594 32456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:24.621919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35605 (local address 127.18.211.67:35825)
0928 06:42:24.622072 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:24.622116 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:24.622768 (+ 652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:24.623074 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:24.623107 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:24.623337 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:42:24.623610 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:24.623616 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:24.624341 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:24.624342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:24.625617 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:24.625620 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:24.626025 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:24.626026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:24.626169 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:24.626698 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:42:24.626709 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:42:24.628473 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:42:24.630226 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:24.630233 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:24.630235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:24.631983 (+ 1748us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:24.632215 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:24.632216 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:24.632216 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:24.632422 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:24.632861 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:42:24.632869 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:42:24.633054 (+ 185us) client_negotiation.cc:769] Sending connection context
0928 06:42:24.633182 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 06:42:24.633353 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260928 06:42:24.634186 32452 heartbeater.cc:347] Connected to a master server at 127.18.211.126:35605
I20260928 06:42:24.634291 32452 heartbeater.cc:464] Registering TS with master...
I20260928 06:42:24.634493 32452 heartbeater.cc:511] Master 127.18.211.126:35605 requested a full tablet report, sending...
I20260928 06:42:24.634744 32186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:24.621896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44702 (local address 127.18.211.126:35605)
0928 06:42:24.621943 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:24.621945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:24.626964 (+ 5019us) server_negotiation.cc:407] Connection header received
0928 06:42:24.627017 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:24.627018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:24.627030 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:24.627106 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:24.627688 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:24.628254 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:24.629321 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:24.629593 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:24.629655 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:24.631635 (+ 1980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:24.631637 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:24.631637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:24.631644 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:24.633156 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:24.634045 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:24.634046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:24.634047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:24.634109 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:24.634299 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:24.634299 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:24.634300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:24.634396 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:24.634550 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:24.634594 (+ 44us) server_negotiation.cc:299] Negotiation successful
0928 06:42:24.634664 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 06:42:24.635304 32452 heartbeater.cc:644] Failed to heartbeat to 127.18.211.126:35605 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 06:42:24.635380 32452 heartbeater.cc:464] Registering TS with master...
I20260928 06:42:24.635447 32452 heartbeater.cc:511] Master 127.18.211.126:35605 requested a full tablet report, sending...
I20260928 06:42:24.635792 32452 heartbeater.cc:464] Registering TS with master...
I20260928 06:42:24.635883 32452 heartbeater.cc:511] Master 127.18.211.126:35605 requested a full tablet report, sending...
W20260928 06:42:24.636199 32452 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 06:42:25 dist-test-slave-b28r krb5kdc[30988](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790577745, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:42:25.267614 32182 heartbeater.cc:464] Registering TS with master...
I20260928 06:42:25.267787 32182 heartbeater.cc:511] Master 127.18.211.126:35605 requested a full tablet report, sending...
I20260928 06:42:25.268154 31984 ts_manager.cc:194] Registered new tserver with Master: 818b03aed0a74de3a6055273ce184e26 (127.18.211.65:36373)
I20260928 06:42:25.269162 31984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.65@KRBTEST.COM'} at 127.18.211.65:42955
I20260928 06:42:25.436290 32316 heartbeater.cc:464] Registering TS with master...
I20260928 06:42:25.436455 32316 heartbeater.cc:511] Master 127.18.211.126:35605 requested a full tablet report, sending...
I20260928 06:42:25.436980 31984 ts_manager.cc:194] Registered new tserver with Master: 6adf7c75f1fe44cb825f8761b44ec495 (127.18.211.66:35049)
I20260928 06:42:25.437474 31984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.66@KRBTEST.COM'} at 127.18.211.66:51933
Sep 28 06:42:25 dist-test-slave-b28r krb5kdc[30988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577745, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for HTTP/127.18.211.84@KRBTEST.COM
I20260928 06:42:25.636379 32452 heartbeater.cc:464] Registering TS with master...
I20260928 06:42:25.636530 32452 heartbeater.cc:511] Master 127.18.211.126:35605 requested a full tablet report, sending...
I20260928 06:42:25.636818 31984 ts_manager.cc:194] Registered new tserver with Master: c6a47d96a5c14791bfdc2e0f58a759c8 (127.18.211.67:44515)
I20260928 06:42:25.637208 31984 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.67@KRBTEST.COM'} at 127.18.211.67:35825
I20260928 06:42:25.645998 19277 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:42:25.654820 32474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:25.647002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44708 (local address 127.18.211.126:35605)
0928 06:42:25.647150 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:25.647153 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:25.647276 (+ 123us) server_negotiation.cc:407] Connection header received
0928 06:42:25.647330 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:25.647331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:25.647346 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:25.647426 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:25.648122 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:25.648890 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:25.649921 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:25.650235 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:25.650313 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:25.651813 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:25.651815 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:25.651816 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:25.651825 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:25.653549 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:25.653967 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:25.653968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:25.653968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:25.654032 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:25.654248 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:25.654250 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:25.654250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:25.654401 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:25.654511 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:25.654624 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 06:42:25.654709 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":97,"threads_started":1}
I20260928 06:42:25.656970 31984 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:42:25.659296 31984 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:42:25.668429 32481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:25.662323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:36373 (local address 127.0.0.1:45016)
0928 06:42:25.663010 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:25.663050 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:25.663136 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:25.663386 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:25.663389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:25.663391 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:42:25.663457 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:25.663458 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:25.665088 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:25.665089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:25.666779 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:25.666782 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:25.668140 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:25.668141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:25.668223 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:25.668236 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 06:42:25.668297 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 06:42:25.668346 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":90,"threads_started":1}
I20260928 06:42:25.668852 32483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:25.662814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45016 (local address 127.18.211.65:36373)
0928 06:42:25.663206 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:25.663211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:25.663223 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:42:25.663273 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:25.663275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:25.663291 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:25.663377 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:42:25.663799 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:25.664997 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:25.666879 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:25.667989 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:25.668048 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:25.668108 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:25.668730 (+ 622us) server_negotiation.cc:299] Negotiation successful
0928 06:42:25.668775 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":65,"threads_started":1}
I20260928 06:42:25.669762 32112 tablet_service.cc:1467] Processing CreateTablet for tablet 83940ba72f78444a8c01b6a622db0898 (DEFAULT_TABLE table=test-table [id=1835876e9fdc4b948039635c91200d82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:42:25.670143 32112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83940ba72f78444a8c01b6a622db0898. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:42:25.671317 32480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:25.662541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:35049 (local address 127.0.0.1:39056)
0928 06:42:25.663605 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:25.663612 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:25.663643 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:25.663943 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:25.663944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:25.663946 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:42:25.664000 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:25.664002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:25.666883 (+ 2881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:25.666884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:25.667982 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:25.667985 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:25.671065 (+ 3080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:25.671067 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:25.671160 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:25.671166 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 06:42:25.671199 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 06:42:25.671246 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1054,"thread_start_us":69,"threads_started":1}
I20260928 06:42:25.671644 32484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:25.662826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39056 (local address 127.18.211.66:35049)
0928 06:42:25.663744 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:25.663749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:25.663758 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:42:25.663789 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:25.663791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:25.663805 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:25.663869 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:42:25.664262 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:25.665304 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:25.668087 (+ 2783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:25.668841 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:25.671423 (+ 2582us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:25.671495 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:25.671523 (+ 28us) server_negotiation.cc:299] Negotiation successful
0928 06:42:25.671567 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":888,"thread_start_us":65,"threads_started":1}
I20260928 06:42:25.672255 32250 tablet_service.cc:1467] Processing CreateTablet for tablet 83940ba72f78444a8c01b6a622db0898 (DEFAULT_TABLE table=test-table [id=1835876e9fdc4b948039635c91200d82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:42:25.672621 32250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83940ba72f78444a8c01b6a622db0898. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:42:25.674741 32485 tablet_bootstrap.cc:492] T 83940ba72f78444a8c01b6a622db0898 P 818b03aed0a74de3a6055273ce184e26: Bootstrap starting.
I20260928 06:42:25.675822 32485 tablet_bootstrap.cc:654] T 83940ba72f78444a8c01b6a622db0898 P 818b03aed0a74de3a6055273ce184e26: Neither blocks nor log segments found. Creating new log.
I20260928 06:42:25.676266 32485 log.cc:824] T 83940ba72f78444a8c01b6a622db0898 P 818b03aed0a74de3a6055273ce184e26: Log is configured to *not* fsync() on all Append() calls
I20260928 06:42:25.676438 32479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:25.662328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:44515 (local address 127.0.0.1:60624)
0928 06:42:25.663488 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:25.663498 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:25.663539 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:25.665417 (+ 1878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:25.665418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:25.665420 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:42:25.665521 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:25.665523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:25.668958 (+ 3435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:25.668959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:25.670094 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:25.670098 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:25.676180 (+ 6082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:25.676181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:25.676288 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:25.676294 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 06:42:25.676330 (+ 36us) client_negotiation.cc:240] Negotiation successful
0928 06:42:25.676369 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1135,"thread_start_us":107,"threads_started":1}
I20260928 06:42:25.676856 32482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:25.662585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60624 (local address 127.18.211.67:44515)
0928 06:42:25.664113 (+ 1528us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:25.664117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:25.664125 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:42:25.664155 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:25.664157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:25.664172 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:25.665594 (+ 1422us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:42:25.665672 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:25.666781 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:25.670200 (+ 3419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:25.670946 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:25.676628 (+ 5682us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:25.676704 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:25.676733 (+ 29us) server_negotiation.cc:299] Negotiation successful
0928 06:42:25.676780 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1494,"thread_start_us":64,"threads_started":1}
I20260928 06:42:25.677350 32485 tablet_bootstrap.cc:492] T 83940ba72f78444a8c01b6a622db0898 P 818b03aed0a74de3a6055273ce184e26: No bootstrap required, opened a new log
I20260928 06:42:25.677412 32485 ts_tablet_manager.cc:1397] T 83940ba72f78444a8c01b6a622db0898 P 818b03aed0a74de3a6055273ce184e26: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:42:25.677455 32385 tablet_service.cc:1467] Processing CreateTablet for tablet 83940ba72f78444a8c01b6a622db0898 (DEFAULT_TABLE table=test-table [id=1835876e9fdc4b948039635c91200d82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:42:25.677742 32385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83940ba72f78444a8c01b6a622db0898. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:42:25.680343 32485 raft_consensus.cc:348] T 83940ba72f78444a8c01b6a622db0898 P 818b03aed0a74de3a6055273ce184e26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a47d96a5c14791bfdc2e0f58a759c8" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 44515 } } peers { permanent_uuid: "818b03aed0a74de3a6055273ce184e26" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 36373 } } peers { permanent_uuid: "6adf7c75f1fe44cb825f8761b44ec495" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 35049 } }
I20260928 06:42:25.680583 32485 raft_consensus.cc:374] T 83940ba72f78444a8c01b6a622db0898 P 818b03aed0a74de3a6055273ce184e26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:42:25.680662 32485 raft_consensus.cc:729] T 83940ba72f78444a8c01b6a622db0898 P 818b03aed0a74de3a6055273ce184e26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 818b03aed0a74de3a6055273ce184e26, State: Initialized, Role: FOLLOWER
I20260928 06:42:25.680861 32485 consensus_queue.cc:260] T 83940ba72f78444a8c01b6a622db0898 P 818b03aed0a74de3a6055273ce184e26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a47d96a5c14791bfdc2e0f58a759c8" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 44515 } } peers { permanent_uuid: "818b03aed0a74de3a6055273ce184e26" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 36373 } } peers { permanent_uuid: "6adf7c75f1fe44cb825f8761b44ec495" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 35049 } }
I20260928 06:42:25.681212 32485 ts_tablet_manager.cc:1428] T 83940ba72f78444a8c01b6a622db0898 P 818b03aed0a74de3a6055273ce184e26: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260928 06:42:25.681255 32182 heartbeater.cc:503] Master 127.18.211.126:35605 was elected leader, sending a full tablet report...
I20260928 06:42:25.681502 32488 tablet_bootstrap.cc:492] T 83940ba72f78444a8c01b6a622db0898 P c6a47d96a5c14791bfdc2e0f58a759c8: Bootstrap starting.
I20260928 06:42:25.682539 32488 tablet_bootstrap.cc:654] T 83940ba72f78444a8c01b6a622db0898 P c6a47d96a5c14791bfdc2e0f58a759c8: Neither blocks nor log segments found. Creating new log.
I20260928 06:42:25.682834 32488 log.cc:824] T 83940ba72f78444a8c01b6a622db0898 P c6a47d96a5c14791bfdc2e0f58a759c8: Log is configured to *not* fsync() on all Append() calls
I20260928 06:42:25.683203 32486 tablet_bootstrap.cc:492] T 83940ba72f78444a8c01b6a622db0898 P 6adf7c75f1fe44cb825f8761b44ec495: Bootstrap starting.
I20260928 06:42:25.683669 32488 tablet_bootstrap.cc:492] T 83940ba72f78444a8c01b6a622db0898 P c6a47d96a5c14791bfdc2e0f58a759c8: No bootstrap required, opened a new log
I20260928 06:42:25.683743 32488 ts_tablet_manager.cc:1397] T 83940ba72f78444a8c01b6a622db0898 P c6a47d96a5c14791bfdc2e0f58a759c8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:42:25.683790 32486 tablet_bootstrap.cc:654] T 83940ba72f78444a8c01b6a622db0898 P 6adf7c75f1fe44cb825f8761b44ec495: Neither blocks nor log segments found. Creating new log.
I20260928 06:42:25.684091 32486 log.cc:824] T 83940ba72f78444a8c01b6a622db0898 P 6adf7c75f1fe44cb825f8761b44ec495: Log is configured to *not* fsync() on all Append() calls
I20260928 06:42:25.686472 32488 raft_consensus.cc:348] T 83940ba72f78444a8c01b6a622db0898 P c6a47d96a5c14791bfdc2e0f58a759c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a47d96a5c14791bfdc2e0f58a759c8" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 44515 } } peers { permanent_uuid: "818b03aed0a74de3a6055273ce184e26" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 36373 } } peers { permanent_uuid: "6adf7c75f1fe44cb825f8761b44ec495" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 35049 } }
I20260928 06:42:25.686630 32488 raft_consensus.cc:374] T 83940ba72f78444a8c01b6a622db0898 P c6a47d96a5c14791bfdc2e0f58a759c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:42:25.686659 32488 raft_consensus.cc:729] T 83940ba72f78444a8c01b6a622db0898 P c6a47d96a5c14791bfdc2e0f58a759c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6a47d96a5c14791bfdc2e0f58a759c8, State: Initialized, Role: FOLLOWER
I20260928 06:42:25.686811 32488 consensus_queue.cc:260] T 83940ba72f78444a8c01b6a622db0898 P c6a47d96a5c14791bfdc2e0f58a759c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a47d96a5c14791bfdc2e0f58a759c8" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 44515 } } peers { permanent_uuid: "818b03aed0a74de3a6055273ce184e26" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 36373 } } peers { permanent_uuid: "6adf7c75f1fe44cb825f8761b44ec495" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 35049 } }
I20260928 06:42:25.687067 32488 ts_tablet_manager.cc:1428] T 83940ba72f78444a8c01b6a622db0898 P c6a47d96a5c14791bfdc2e0f58a759c8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:42:25.687121 32452 heartbeater.cc:503] Master 127.18.211.126:35605 was elected leader, sending a full tablet report...
I20260928 06:42:25.691761 32486 tablet_bootstrap.cc:492] T 83940ba72f78444a8c01b6a622db0898 P 6adf7c75f1fe44cb825f8761b44ec495: No bootstrap required, opened a new log
I20260928 06:42:25.691841 32486 ts_tablet_manager.cc:1397] T 83940ba72f78444a8c01b6a622db0898 P 6adf7c75f1fe44cb825f8761b44ec495: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260928 06:42:25.694079 32486 raft_consensus.cc:348] T 83940ba72f78444a8c01b6a622db0898 P 6adf7c75f1fe44cb825f8761b44ec495 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a47d96a5c14791bfdc2e0f58a759c8" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 44515 } } peers { permanent_uuid: "818b03aed0a74de3a6055273ce184e26" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 36373 } } peers { permanent_uuid: "6adf7c75f1fe44cb825f8761b44ec495" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 35049 } }
I20260928 06:42:25.694248 32486 raft_consensus.cc:374] T 83940ba72f78444a8c01b6a622db0898 P 6adf7c75f1fe44cb825f8761b44ec495 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:42:25.694289 32486 raft_consensus.cc:729] T 83940ba72f78444a8c01b6a622db0898 P 6adf7c75f1fe44cb825f8761b44ec495 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6adf7c75f1fe44cb825f8761b44ec495, State: Initialized, Role: FOLLOWER
I20260928 06:42:25.694409 32486 consensus_queue.cc:260] T 83940ba72f78444a8c01b6a622db0898 P 6adf7c75f1fe44cb825f8761b44ec495 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a47d96a5c14791bfdc2e0f58a759c8" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 44515 } } peers { permanent_uuid: "818b03aed0a74de3a6055273ce184e26" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 36373 } } peers { permanent_uuid: "6adf7c75f1fe44cb825f8761b44ec495" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 35049 } }
I20260928 06:42:25.694700 32486 ts_tablet_manager.cc:1428] T 83940ba72f78444a8c01b6a622db0898 P 6adf7c75f1fe44cb825f8761b44ec495: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:42:25.694722 32316 heartbeater.cc:503] Master 127.18.211.126:35605 was elected leader, sending a full tablet report...
I20260928 06:42:25.734354 32492 raft_consensus.cc:482] T 83940ba72f78444a8c01b6a622db0898 P c6a47d96a5c14791bfdc2e0f58a759c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:42:25.734567 32492 raft_consensus.cc:504] T 83940ba72f78444a8c01b6a622db0898 P c6a47d96a5c14791bfdc2e0f58a759c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a47d96a5c14791bfdc2e0f58a759c8" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 44515 } } peers { permanent_uuid: "818b03aed0a74de3a6055273ce184e26" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 36373 } } peers { permanent_uuid: "6adf7c75f1fe44cb825f8761b44ec495" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 35049 } }
I20260928 06:42:25.735138 32492 leader_election.cc:288] T 83940ba72f78444a8c01b6a622db0898 P c6a47d96a5c14791bfdc2e0f58a759c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 818b03aed0a74de3a6055273ce184e26 (127.18.211.65:36373), 6adf7c75f1fe44cb825f8761b44ec495 (127.18.211.66:35049)
I20260928 06:42:25.739972 32483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:25.735553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:43327 (local address 127.18.211.65:36373)
0928 06:42:25.735604 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:25.735606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:25.736269 (+ 663us) server_negotiation.cc:407] Connection header received
0928 06:42:25.736388 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:25.736389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:25.736399 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:25.736428 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:42:25.736677 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:25.737396 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:25.738977 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:25.739559 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:25.739610 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:25.739619 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:25.739854 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 06:42:25.739897 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:42:25.740440 32494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:25.735443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:36373 (local address 127.18.211.67:43327)
0928 06:42:25.736183 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:25.736194 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:25.736305 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:25.736514 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:25.736514 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:25.736517 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:42:25.736598 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:25.736600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:25.737499 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:25.737500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:25.738864 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:25.738867 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:25.739695 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:25.739696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:25.739765 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:25.739769 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:42:25.740300 (+ 531us) client_negotiation.cc:240] Negotiation successful
0928 06:42:25.740345 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":111,"thread_start_us":115,"threads_started":1}
I20260928 06:42:25.740947 32120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83940ba72f78444a8c01b6a622db0898" candidate_uuid: "c6a47d96a5c14791bfdc2e0f58a759c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818b03aed0a74de3a6055273ce184e26" is_pre_election: true
I20260928 06:42:25.741123 32120 raft_consensus.cc:2454] T 83940ba72f78444a8c01b6a622db0898 P 818b03aed0a74de3a6055273ce184e26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6a47d96a5c14791bfdc2e0f58a759c8 in term 0.
I20260928 06:42:25.741459 32336 leader_election.cc:302] T 83940ba72f78444a8c01b6a622db0898 P c6a47d96a5c14791bfdc2e0f58a759c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818b03aed0a74de3a6055273ce184e26, c6a47d96a5c14791bfdc2e0f58a759c8; no voters:
I20260928 06:42:25.741488 32484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:25.735855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:41245 (local address 127.18.211.66:35049)
0928 06:42:25.735923 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:25.735925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:25.736355 (+ 430us) server_negotiation.cc:407] Connection header received
0928 06:42:25.736456 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:25.736457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:25.736467 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:25.736493 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:42:25.738504 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:25.739233 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:25.740530 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:25.741048 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:25.741092 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:25.741099 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:25.741378 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 06:42:25.741423 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:42:25.741709 32495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:25.735660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:35049 (local address 127.18.211.67:41245)
0928 06:42:25.736292 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:25.736303 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:25.736386 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:25.736554 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:25.736555 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:25.736557 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:42:25.737568 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:25.737569 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:25.739328 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:25.739328 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:25.740436 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:25.740440 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:25.741154 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:25.741155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:25.741253 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:25.741257 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:42:25.741600 (+ 343us) client_negotiation.cc:240] Negotiation successful
0928 06:42:25.741640 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":485,"threads_started":1}
I20260928 06:42:25.741724 32492 raft_consensus.cc:2781] T 83940ba72f78444a8c01b6a622db0898 P c6a47d96a5c14791bfdc2e0f58a759c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:42:25.742123 32270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83940ba72f78444a8c01b6a622db0898" candidate_uuid: "c6a47d96a5c14791bfdc2e0f58a759c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6adf7c75f1fe44cb825f8761b44ec495" is_pre_election: true
I20260928 06:42:25.742286 32270 raft_consensus.cc:2454] T 83940ba72f78444a8c01b6a622db0898 P 6adf7c75f1fe44cb825f8761b44ec495 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6a47d96a5c14791bfdc2e0f58a759c8 in term 0.
I20260928 06:42:25.741976 32492 raft_consensus.cc:482] T 83940ba72f78444a8c01b6a622db0898 P c6a47d96a5c14791bfdc2e0f58a759c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:42:25.742456 32492 raft_consensus.cc:3037] T 83940ba72f78444a8c01b6a622db0898 P c6a47d96a5c14791bfdc2e0f58a759c8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:42:25.743479 32492 raft_consensus.cc:504] T 83940ba72f78444a8c01b6a622db0898 P c6a47d96a5c14791bfdc2e0f58a759c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a47d96a5c14791bfdc2e0f58a759c8" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 44515 } } peers { permanent_uuid: "818b03aed0a74de3a6055273ce184e26" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 36373 } } peers { permanent_uuid: "6adf7c75f1fe44cb825f8761b44ec495" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 35049 } }
I20260928 06:42:25.743618 32492 leader_election.cc:288] T 83940ba72f78444a8c01b6a622db0898 P c6a47d96a5c14791bfdc2e0f58a759c8 [CANDIDATE]: Term 1 election: Requested vote from peers 818b03aed0a74de3a6055273ce184e26 (127.18.211.65:36373), 6adf7c75f1fe44cb825f8761b44ec495 (127.18.211.66:35049)
I20260928 06:42:25.743795 32270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83940ba72f78444a8c01b6a622db0898" candidate_uuid: "c6a47d96a5c14791bfdc2e0f58a759c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6adf7c75f1fe44cb825f8761b44ec495"
I20260928 06:42:25.743821 32120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83940ba72f78444a8c01b6a622db0898" candidate_uuid: "c6a47d96a5c14791bfdc2e0f58a759c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "818b03aed0a74de3a6055273ce184e26"
I20260928 06:42:25.743865 32270 raft_consensus.cc:3037] T 83940ba72f78444a8c01b6a622db0898 P 6adf7c75f1fe44cb825f8761b44ec495 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:42:25.743916 32120 raft_consensus.cc:3037] T 83940ba72f78444a8c01b6a622db0898 P 818b03aed0a74de3a6055273ce184e26 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:42:25.744699 32270 raft_consensus.cc:2454] T 83940ba72f78444a8c01b6a622db0898 P 6adf7c75f1fe44cb825f8761b44ec495 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6a47d96a5c14791bfdc2e0f58a759c8 in term 1.
I20260928 06:42:25.744776 32120 raft_consensus.cc:2454] T 83940ba72f78444a8c01b6a622db0898 P 818b03aed0a74de3a6055273ce184e26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6a47d96a5c14791bfdc2e0f58a759c8 in term 1.
I20260928 06:42:25.744943 32336 leader_election.cc:302] T 83940ba72f78444a8c01b6a622db0898 P c6a47d96a5c14791bfdc2e0f58a759c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 818b03aed0a74de3a6055273ce184e26, c6a47d96a5c14791bfdc2e0f58a759c8; no voters:
I20260928 06:42:25.745044 32492 raft_consensus.cc:2781] T 83940ba72f78444a8c01b6a622db0898 P c6a47d96a5c14791bfdc2e0f58a759c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:42:25.745208 32492 raft_consensus.cc:686] T 83940ba72f78444a8c01b6a622db0898 P c6a47d96a5c14791bfdc2e0f58a759c8 [term 1 LEADER]: Becoming Leader. State: Replica: c6a47d96a5c14791bfdc2e0f58a759c8, State: Running, Role: LEADER
I20260928 06:42:25.745376 32492 consensus_queue.cc:237] T 83940ba72f78444a8c01b6a622db0898 P c6a47d96a5c14791bfdc2e0f58a759c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a47d96a5c14791bfdc2e0f58a759c8" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 44515 } } peers { permanent_uuid: "818b03aed0a74de3a6055273ce184e26" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 36373 } } peers { permanent_uuid: "6adf7c75f1fe44cb825f8761b44ec495" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 35049 } }
I20260928 06:42:25.746219 31984 catalog_manager.cc:5495] T 83940ba72f78444a8c01b6a622db0898 P c6a47d96a5c14791bfdc2e0f58a759c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6a47d96a5c14791bfdc2e0f58a759c8 (127.18.211.67). New cstate: current_term: 1 leader_uuid: "c6a47d96a5c14791bfdc2e0f58a759c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6a47d96a5c14791bfdc2e0f58a759c8" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 44515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "818b03aed0a74de3a6055273ce184e26" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 36373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6adf7c75f1fe44cb825f8761b44ec495" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 35049 } health_report { overall_health: UNKNOWN } } }
W20260928 06:42:25.749136 32183 tablet.cc:2369] T 83940ba72f78444a8c01b6a622db0898 P 818b03aed0a74de3a6055273ce184e26: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:42:25.760075 32482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:25.756908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60634 (local address 127.18.211.67:44515)
0928 06:42:25.756950 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:25.756952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:25.756963 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:42:25.757017 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:25.757018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:25.757037 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:25.757105 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:42:25.757247 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:25.757994 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:25.759005 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:25.759320 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:25.759390 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:25.759611 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:42:25.759739 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:42:25.759821 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:25.759983 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 06:42:25.760003 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:42:25.763808 32270 raft_consensus.cc:1261] T 83940ba72f78444a8c01b6a622db0898 P 6adf7c75f1fe44cb825f8761b44ec495 [term 1 FOLLOWER]: Refusing update from remote peer c6a47d96a5c14791bfdc2e0f58a759c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:42:25.764168 32492 consensus_queue.cc:1035] T 83940ba72f78444a8c01b6a622db0898 P c6a47d96a5c14791bfdc2e0f58a759c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6adf7c75f1fe44cb825f8761b44ec495" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 35049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:42:25.764753 32120 raft_consensus.cc:1261] T 83940ba72f78444a8c01b6a622db0898 P 818b03aed0a74de3a6055273ce184e26 [term 1 FOLLOWER]: Refusing update from remote peer c6a47d96a5c14791bfdc2e0f58a759c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:42:25.765244 32492 consensus_queue.cc:1035] T 83940ba72f78444a8c01b6a622db0898 P c6a47d96a5c14791bfdc2e0f58a759c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "818b03aed0a74de3a6055273ce184e26" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 36373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:42:25.768462 32500 mvcc.cc:205] Tried to move back new op lower bound from 7334206446646013952 to 7334206446573756416. Current Snapshot: MvccSnapshot[applied={T|T < 7334206446646013952}]
I20260928 06:42:25.770993 31983 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:42:25.771095 32499 mvcc.cc:205] Tried to move back new op lower bound from 7334206446646013952 to 7334206446573756416. Current Snapshot: MvccSnapshot[applied={T|T < 7334206446646013952 or (T in {7334206446646013952})}]
I20260928 06:42:25.771117 31983 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:42:25.771756 32502 mvcc.cc:205] Tried to move back new op lower bound from 7334206446646013952 to 7334206446573756416. Current Snapshot: MvccSnapshot[applied={T|T < 7334206446646013952}]
I20260928 06:42:25.771821 31983 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1a6ca4f004944f2a90147b9fae23596: Sending DeleteTablet for 3 replicas of tablet 83940ba72f78444a8c01b6a622db0898
I20260928 06:42:25.772450 32385 tablet_service.cc:1514] Processing DeleteTablet for tablet 83940ba72f78444a8c01b6a622db0898 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:42:25 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:60624
I20260928 06:42:25.772675 32112 tablet_service.cc:1514] Processing DeleteTablet for tablet 83940ba72f78444a8c01b6a622db0898 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:42:25 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:45016
I20260928 06:42:25.772907 32250 tablet_service.cc:1514] Processing DeleteTablet for tablet 83940ba72f78444a8c01b6a622db0898 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:42:25 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:39056
I20260928 06:42:25.773478 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 32023
I20260928 06:42:25.777915 32510 tablet_replica.cc:330] T 83940ba72f78444a8c01b6a622db0898 P 6adf7c75f1fe44cb825f8761b44ec495: stopping tablet replica
I20260928 06:42:25.778069 32508 tablet_replica.cc:330] T 83940ba72f78444a8c01b6a622db0898 P c6a47d96a5c14791bfdc2e0f58a759c8: stopping tablet replica
I20260928 06:42:25.778260 32508 raft_consensus.cc:2229] T 83940ba72f78444a8c01b6a622db0898 P c6a47d96a5c14791bfdc2e0f58a759c8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:42:25.778414 32510 raft_consensus.cc:2229] T 83940ba72f78444a8c01b6a622db0898 P 6adf7c75f1fe44cb825f8761b44ec495 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:42:25.778568 32508 raft_consensus.cc:2258] T 83940ba72f78444a8c01b6a622db0898 P c6a47d96a5c14791bfdc2e0f58a759c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:42:25.778743 32510 raft_consensus.cc:2258] T 83940ba72f78444a8c01b6a622db0898 P 6adf7c75f1fe44cb825f8761b44ec495 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:42:25.779356 32508 ts_tablet_manager.cc:1905] T 83940ba72f78444a8c01b6a622db0898 P c6a47d96a5c14791bfdc2e0f58a759c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:42:25.779826 32510 ts_tablet_manager.cc:1905] T 83940ba72f78444a8c01b6a622db0898 P 6adf7c75f1fe44cb825f8761b44ec495: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:42:25.784080 32510 ts_tablet_manager.cc:1918] T 83940ba72f78444a8c01b6a622db0898 P 6adf7c75f1fe44cb825f8761b44ec495: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:42:25.784260 32508 ts_tablet_manager.cc:1918] T 83940ba72f78444a8c01b6a622db0898 P c6a47d96a5c14791bfdc2e0f58a759c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:42:25.784365 32508 log.cc:1196] T 83940ba72f78444a8c01b6a622db0898 P c6a47d96a5c14791bfdc2e0f58a759c8: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-2/wal/wals/83940ba72f78444a8c01b6a622db0898
I20260928 06:42:25.784711 32508 ts_tablet_manager.cc:1939] T 83940ba72f78444a8c01b6a622db0898 P c6a47d96a5c14791bfdc2e0f58a759c8: Deleting consensus metadata
I20260928 06:42:25.784857 32510 log.cc:1196] T 83940ba72f78444a8c01b6a622db0898 P 6adf7c75f1fe44cb825f8761b44ec495: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790577602522428-19277-0/minicluster-data/ts-1/wal/wals/83940ba72f78444a8c01b6a622db0898
I20260928 06:42:25.785203 32510 ts_tablet_manager.cc:1939] T 83940ba72f78444a8c01b6a622db0898 P 6adf7c75f1fe44cb825f8761b44ec495: Deleting consensus metadata
I20260928 06:42:25.785729 31971 catalog_manager.cc:4841] TS c6a47d96a5c14791bfdc2e0f58a759c8 (127.18.211.67:44515): tablet 83940ba72f78444a8c01b6a622db0898 (table test-table [id=1835876e9fdc4b948039635c91200d82]) successfully deleted
I20260928 06:42:25.786278 31968 catalog_manager.cc:4841] TS 6adf7c75f1fe44cb825f8761b44ec495 (127.18.211.66:35049): tablet 83940ba72f78444a8c01b6a622db0898 (table test-table [id=1835876e9fdc4b948039635c91200d82]) successfully deleted
W20260928 06:42:25.789924 31968 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.211.65:36373 (error 108)
I20260928 06:42:25.790884 32479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:25.790708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:36373 (local address 127.0.0.1:45018)
0928 06:42:25.790791 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:25.790827 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.65:36373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 06:42:25.791114 31968 catalog_manager.cc:4584] TS 818b03aed0a74de3a6055273ce184e26 (127.18.211.65:36373): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 83940ba72f78444a8c01b6a622db0898: Network error: Client connection negotiation failed: client connection to 127.18.211.65:36373: connect: Connection refused (error 111)
I20260928 06:42:25.791376 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 32184
I20260928 06:42:25.799428 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 32319
I20260928 06:42:25.804530 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 31951
I20260928 06:42:25.809626 19277 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 06:42:26.143587 19277 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 06:42:26.143679 19277 mini_ranger.cc:66] Stopping Ranger...
I20260928 06:42:26.478233 19277 mini_ranger.cc:68] Stopped Ranger
2026-09-28 06:42:26.478 UTC [31018] LOG: received smart shutdown request
2026-09-28 06:42:26.481 UTC [31018] LOG: background worker "logical replication launcher" (PID 31027) exited with exit code 1
2026-09-28 06:42:26.482 UTC [31022] LOG: shutting down
2026-09-28 06:42:26.632 UTC [31018] LOG: database system is shut down
2026-09-28T06:42:26Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46668 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:42:26 dist-test-slave-b28r krb5kdc[32521](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:42:26 dist-test-slave-b28r krb5kdc[32521](info): set up 2 sockets
Sep 28 06:42:26 dist-test-slave-b28r krb5kdc[32521](info): commencing operation
krb5kdc: starting...
W20260928 06:42:27.679797 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:42:27 dist-test-slave-b28r krb5kdc[32521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:42:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:42:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.211.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
I20260928 06:42:27.787837 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:35959
--webserver_interface=127.18.211.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:40709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:35959,127.18.211.125:34665,127.18.211.124:36273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:42:27.867712 32534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:42:27.867941 32534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:42:27.867964 32534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:42:27.869107 32534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:42:27.869148 32534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:42:27.869160 32534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:42:27.869171 32534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:42:27.869189 32534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:42:27.870663 32534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:40709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:35959,127.18.211.125:34665,127.18.211.124:36273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:35959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:42:27.870900 32534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:42:27.871141 32534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:42:27.875653 32542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:27.875658 32540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:27.875865 32539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:42:27.875873 32534 server_base.cc:1034] running on GCE node
I20260928 06:42:27.876104 32534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:42:27.876375 32534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:42:27.877519 32534 hybrid_clock.cc:648] HybridClock initialized: now 1790577747877507 us; error 27 us; skew 500 ppm
Sep 28 06:42:27 dist-test-slave-b28r krb5kdc[32521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577747, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:42:27.879901 32534 init.cc:377] Logged in from keytab as kudu/127.18.211.126@KRBTEST.COM (short username kudu)
I20260928 06:42:27.880373 32534 webserver.cc:460] Webserver started at http://127.18.211.126:46607/ using document root <none> and password file <none>
I20260928 06:42:27.880584 32534 fs_manager.cc:359] Metadata directory not provided
I20260928 06:42:27.880628 32534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:42:27.880746 32534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:42:27.881784 32534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "87e3ca67f1d14df5b0921366d43c4b09"
format_stamp: "Formatted at 2026-09-28 06:42:27 on dist-test-slave-b28r"
server_key: "c33b9c71a52b7613e05853ab3902c8a4"
server_key_iv: "fc399ba1783ed697214a6acbaba61d85"
server_key_version: "encryptionkey@0"
I20260928 06:42:27.882104 32534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "87e3ca67f1d14df5b0921366d43c4b09"
format_stamp: "Formatted at 2026-09-28 06:42:27 on dist-test-slave-b28r"
server_key: "c33b9c71a52b7613e05853ab3902c8a4"
server_key_iv: "fc399ba1783ed697214a6acbaba61d85"
server_key_version: "encryptionkey@0"
I20260928 06:42:27.883347 32534 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:42:27.883960 32549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:27.884099 32534 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:42:27.884184 32534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "87e3ca67f1d14df5b0921366d43c4b09"
format_stamp: "Formatted at 2026-09-28 06:42:27 on dist-test-slave-b28r"
server_key: "c33b9c71a52b7613e05853ab3902c8a4"
server_key_iv: "fc399ba1783ed697214a6acbaba61d85"
server_key_version: "encryptionkey@0"
I20260928 06:42:27.884235 32534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:42:27.898906 32534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:42:27.900799 32534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:42:27.900924 32534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:42:27.904677 32534 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:35959
I20260928 06:42:27.904729 32601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:35959 every 8 connection(s)
I20260928 06:42:27.905028 32534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:42:27.905961 32602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:42:27.907521 32602 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35959 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:42:27.912174 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 32534
I20260928 06:42:27.912277 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:42:27.912426 19277 external_mini_cluster.cc:1428] Setting key e911b65b8f015c39ca7279811328e28e
Sep 28 06:42:27 dist-test-slave-b28r krb5kdc[32521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577747, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:42:27.917059 32604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:27.908162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35959 (local address 127.0.0.1:33474)
0928 06:42:27.908298 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:27.908335 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:27.908769 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:27.909048 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:27.909051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:27.909322 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:42:27.909475 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:27.909480 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:27.910407 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:27.910408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:27.911158 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:27.911168 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:27.911728 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:27.911729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:27.911799 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:27.912429 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:42:27.912437 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:42:27.913847 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:42:27.915663 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:27.915669 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:27.915671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:27.915881 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:27.916151 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:27.916152 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:27.916152 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:27.916237 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:27.916643 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:42:27.916647 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:42:27.916701 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 06:42:27.916784 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 06:42:27.916904 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260928 06:42:27.917083 32605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:27.908322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33474 (local address 127.18.211.126:35959)
0928 06:42:27.908434 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:27.908438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:27.908465 (+ 27us) server_negotiation.cc:407] Connection header received
0928 06:42:27.908887 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:27.908888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:27.908904 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:27.908976 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:27.909544 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:27.910333 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:27.911240 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:27.911571 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:27.911645 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:27.913934 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:27.913942 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:27.913942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:27.913949 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:27.915570 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:27.915972 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:27.915975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:27.915976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:27.916085 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:27.916299 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:27.916300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:27.916300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:27.916573 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:27.916655 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:27.916838 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 06:42:27.916982 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":86,"thread_start_us":41,"threads_started":1}
I20260928 06:42:27.917939 32602 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.211.125" port: 34665 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:42:27.918256 32604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:27.918148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.125:34665 (local address 127.0.0.1:50476)
0928 06:42:27.918189 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:27.918237 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.125:34665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 06:42:27.918380 32550 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.211.125:34665: connect: Connection refused (error 111)
I20260928 06:42:27.918781 32604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:27.918694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.125:34665 (local address 127.0.0.1:50482)
0928 06:42:27.918734 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:27.918759 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.125:34665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 06:42:27.918902 32602 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.211.125:34665: Network error: Client connection negotiation failed: client connection to 127.18.211.125:34665: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.18.211.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.125.keytab.
Entry for principal kudu/127.18.211.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.125.keytab.
I20260928 06:42:27.936969 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.125:34665
--webserver_interface=127.18.211.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.125.keytab
--principal=kudu/127.18.211.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:40709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:35959,127.18.211.125:34665,127.18.211.124:36273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:42:27.968094 32602 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.211.125" port: 34665 } attempt: 1
I20260928 06:42:27.968628 32604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:27.968454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.125:34665 (local address 127.0.0.1:50496)
0928 06:42:27.968532 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:27.968586 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.125:34665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 06:42:27.968997 32604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:27.968907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.125:34665 (local address 127.0.0.1:50512)
0928 06:42:27.968944 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:27.968974 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.125:34665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 06:42:27.969112 32602 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.211.125:34665: Network error: Client connection negotiation failed: client connection to 127.18.211.125:34665: connect: Connection refused (error 111)
W20260928 06:42:28.016567 32609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:42:28.016772 32609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:42:28.016791 32609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:42:28.017904 32609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:42:28.017942 32609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:42:28.017956 32609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:42:28.017966 32609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:42:28.017979 32609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:42:28.019435 32609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:40709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:35959,127.18.211.125:34665,127.18.211.124:36273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.125.keytab
--principal=kudu/127.18.211.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.125:34665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.211.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:42:28.019649 32609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:42:28.019858 32609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:42:28.024317 32617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:28.024327 32614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:28.024315 32615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:42:28.024502 32609 server_base.cc:1034] running on GCE node
I20260928 06:42:28.024806 32609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:42:28.025087 32609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:42:28.026286 32609 hybrid_clock.cc:648] HybridClock initialized: now 1790577748026269 us; error 34 us; skew 500 ppm
Sep 28 06:42:28 dist-test-slave-b28r krb5kdc[32521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577748, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:42:28.028678 32609 init.cc:377] Logged in from keytab as kudu/127.18.211.125@KRBTEST.COM (short username kudu)
I20260928 06:42:28.029088 32609 webserver.cc:460] Webserver started at http://127.18.211.125:34703/ using document root <none> and password file <none>
I20260928 06:42:28.029278 32609 fs_manager.cc:359] Metadata directory not provided
I20260928 06:42:28.029320 32609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:42:28.029439 32609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:42:28.030463 32609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-1/data/instance:
uuid: "7dce2a529ac646a89a2b9423cb735bcd"
format_stamp: "Formatted at 2026-09-28 06:42:28 on dist-test-slave-b28r"
server_key: "3df731271839c4a953882cd88a32bbcb"
server_key_iv: "25dc43a79e134e9e575937b09d50e576"
server_key_version: "encryptionkey@0"
I20260928 06:42:28.030774 32609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-1/wal/instance:
uuid: "7dce2a529ac646a89a2b9423cb735bcd"
format_stamp: "Formatted at 2026-09-28 06:42:28 on dist-test-slave-b28r"
server_key: "3df731271839c4a953882cd88a32bbcb"
server_key_iv: "25dc43a79e134e9e575937b09d50e576"
server_key_version: "encryptionkey@0"
I20260928 06:42:28.032024 32609 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:42:28.032727 32624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:28.032852 32609 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:42:28.032930 32609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-1/wal
uuid: "7dce2a529ac646a89a2b9423cb735bcd"
format_stamp: "Formatted at 2026-09-28 06:42:28 on dist-test-slave-b28r"
server_key: "3df731271839c4a953882cd88a32bbcb"
server_key_iv: "25dc43a79e134e9e575937b09d50e576"
server_key_version: "encryptionkey@0"
I20260928 06:42:28.032989 32609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:42:28.037269 32602 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.211.125" port: 34665 } attempt: 2
I20260928 06:42:28.037698 32604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:28.037566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.125:34665 (local address 127.0.0.1:50524)
0928 06:42:28.037623 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:28.037661 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.125:34665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 06:42:28.038023 32604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:28.037952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.125:34665 (local address 127.0.0.1:50536)
0928 06:42:28.037985 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:28.038006 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.125:34665: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260928 06:42:28.038107 32602 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.211.125:34665: Network error: Client connection negotiation failed: client connection to 127.18.211.125:34665: connect: Connection refused (error 111)
I20260928 06:42:28.053337 32609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:42:28.055564 32609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:42:28.055701 32609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:42:28.059664 32609 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.125:34665
I20260928 06:42:28.059705 32677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.125:34665 every 8 connection(s)
I20260928 06:42:28.060043 32609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-1/data/info.pb
I20260928 06:42:28.060878 32678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:42:28.061448 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 32609
I20260928 06:42:28.061568 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-1/wal/instance
I20260928 06:42:28.061750 19277 external_mini_cluster.cc:1428] Setting key 17dd1b0d3213ee8379a206f2a01891e1
I20260928 06:42:28.063020 32678 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35959 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.18.211.124@KRBTEST.COM; defaulting to no policy
Sep 28 06:42:28 dist-test-slave-b28r krb5kdc[32521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577748, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.125@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.124@KRBTEST.COM" created.
I20260928 06:42:28.073335 32605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:28.063859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33476 (local address 127.18.211.126:35959)
0928 06:42:28.063919 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:28.063921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:28.064179 (+ 258us) server_negotiation.cc:407] Connection header received
0928 06:42:28.064774 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:28.064775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:28.064790 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:28.064845 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:28.065679 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.066162 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.067647 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.067846 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.067910 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:28.070303 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:28.070304 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:28.070305 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:28.070311 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:28.071656 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:28.072231 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:28.072232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:28.072233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:28.072294 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:28.072670 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:28.072671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:28.072672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:28.072796 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:28.072857 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:28.073184 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 06:42:28.073268 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 06:42:28.073568 32681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:28.063786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35959 (local address 127.0.0.1:33476)
0928 06:42:28.064002 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:28.064056 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:28.064666 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:28.064916 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:28.064920 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:28.065263 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:42:28.065579 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:28.065586 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.066258 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.066259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:28.067567 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:28.067570 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.067937 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.067939 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:28.068134 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:28.068647 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:42:28.068657 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:42:28.070178 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:42:28.071748 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:28.071756 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:28.071758 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:28.072064 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:28.072412 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:28.072413 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:28.072414 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:28.072570 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:28.072864 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:42:28.072871 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:42:28.073037 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 06:42:28.073161 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 06:42:28.073335 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":79,"threads_started":1}
I20260928 06:42:28.074314 32678 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.211.125" port: 34665 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:42:28 dist-test-slave-b28r krb5kdc[32521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577748, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.125@KRBTEST.COM for kudu/127.18.211.125@KRBTEST.COM
I20260928 06:42:28.081228 32681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:28.074588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.125:34665 (local address 127.0.0.1:50550)
0928 06:42:28.074639 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:28.074651 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:28.074696 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:28.074921 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:28.074922 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:28.075065 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:42:28.075168 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:28.075171 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.076071 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.076072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:28.076567 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:28.076568 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.077005 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.077006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:28.077069 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:28.077460 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:42:28.077465 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:42:28.078584 (+ 1119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:42:28.080223 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:28.080224 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:28.080224 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:28.080399 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:28.080593 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:28.080593 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:28.080594 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:28.080653 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:28.080966 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:42:28.080967 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:42:28.081009 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 06:42:28.081072 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 06:42:28.081164 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 06:42:28.081368 32683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:28.074674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50550 (local address 127.18.211.125:34665)
0928 06:42:28.074803 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:28.074807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:28.074822 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:42:28.074837 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:28.074838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:28.074849 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:28.074911 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:28.075246 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.075986 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.076648 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.076938 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.077002 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:28.078673 (+ 1671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:28.078682 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:28.078683 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:28.078690 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:28.080109 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:28.080481 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:28.080484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:28.080485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:28.080538 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:28.080742 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:28.080744 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:28.080744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:28.080891 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:28.080963 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:28.081132 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 06:42:28.081252 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
I20260928 06:42:28.081995 32678 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.211.124" port: 36273 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:42:28.082337 32681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:28.082189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:36273 (local address 127.0.0.1:40622)
0928 06:42:28.082271 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:28.082317 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.124:36273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 06:42:28.082486 32625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.211.124:36273: connect: Connection refused (error 111)
I20260928 06:42:28.082983 32681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:28.082888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:36273 (local address 127.0.0.1:40630)
0928 06:42:28.082931 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:28.082962 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.124:36273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 06:42:28.083134 32678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.211.124:36273: Network error: Client connection negotiation failed: client connection to 127.18.211.124:36273: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.124.keytab.
Entry for principal kudu/127.18.211.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.124.keytab.
I20260928 06:42:28.085717 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.124:36273
--webserver_interface=127.18.211.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.124.keytab
--principal=kudu/127.18.211.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:40709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:35959,127.18.211.125:34665,127.18.211.124:36273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:42:28.129258 32602 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.211.125" port: 34665 } attempt: 3
I20260928 06:42:28.132395 32678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.211.124" port: 36273 } attempt: 1
I20260928 06:42:28.132750 32681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:28.132654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:36273 (local address 127.0.0.1:40646)
0928 06:42:28.132691 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:28.132719 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.124:36273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260928 06:42:28.133095 32681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:28.133009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:36273 (local address 127.0.0.1:40660)
0928 06:42:28.133049 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:28.133073 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.124:36273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 06:42:28.133162 32678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.211.124:36273: Network error: Client connection negotiation failed: client connection to 127.18.211.124:36273: connect: Connection refused (error 111)
Sep 28 06:42:28 dist-test-slave-b28r krb5kdc[32521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577747, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for kudu/127.18.211.125@KRBTEST.COM
I20260928 06:42:28.135941 32683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:28.129674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50556 (local address 127.18.211.125:34665)
0928 06:42:28.129735 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:28.129738 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:28.129748 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:42:28.129784 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:28.129785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:28.129798 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:28.129843 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:28.130150 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.130668 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.131381 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.131594 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.131650 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:28.133630 (+ 1980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:28.133631 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:28.133631 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:28.133639 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:28.134889 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:28.135254 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:28.135256 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:28.135256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:28.135313 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:28.135476 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:28.135477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:28.135478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:28.135583 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:28.135636 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:28.135772 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 06:42:28.135852 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 06:42:28.135946 32604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:28.129605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.125:34665 (local address 127.0.0.1:50556)
0928 06:42:28.129657 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:28.129668 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:28.129716 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:28.129855 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:28.129857 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:28.130011 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:42:28.130077 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:28.130079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.130752 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.130753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:28.131309 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:28.131312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.131656 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.131657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:28.131728 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:28.132132 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:42:28.132138 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:42:28.133543 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:42:28.134967 (+ 1424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:28.134968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:28.134968 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:28.135185 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:28.135366 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:28.135367 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:28.135367 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:28.135426 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:28.135636 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:42:28.135637 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:42:28.135686 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 06:42:28.135744 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 06:42:28.135850 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 06:42:28.136746 32602 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.211.124" port: 36273 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:42:28.137068 32604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:28.136974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:36273 (local address 127.0.0.1:40662)
0928 06:42:28.137016 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:28.137042 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.124:36273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 06:42:28.137321 32604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:28.137246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:36273 (local address 127.0.0.1:40678)
0928 06:42:28.137282 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:28.137303 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.124:36273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 06:42:28.137423 32602 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.211.124:36273: Network error: Client connection negotiation failed: client connection to 127.18.211.124:36273: connect: Connection refused (error 111)
W20260928 06:42:28.168181 32685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:42:28.168391 32685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:42:28.168422 32685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 06:42:28.168576 32602 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.211.124" port: 36273 } attempt: 1
I20260928 06:42:28.168993 32604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:28.168870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:36273 (local address 127.0.0.1:40690)
0928 06:42:28.168927 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:28.168963 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.124:36273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 06:42:28.169341 32604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:28.169244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:36273 (local address 127.0.0.1:40706)
0928 06:42:28.169285 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:28.169310 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.124:36273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 06:42:28.169422 32602 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.211.124:36273: Network error: Client connection negotiation failed: client connection to 127.18.211.124:36273: connect: Connection refused (error 111)
W20260928 06:42:28.169691 32685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:42:28.169734 32685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:42:28.169747 32685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:42:28.169765 32685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:42:28.169782 32685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:42:28.171276 32685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:40709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:35959,127.18.211.125:34665,127.18.211.124:36273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.124.keytab
--principal=kudu/127.18.211.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.124:36273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.211.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:42:28.171474 32685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:42:28.171650 32685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:42:28.176369 32693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:42:28.176385 32685 server_base.cc:1034] running on GCE node
W20260928 06:42:28.176389 32690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:28.176362 32691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:42:28.176887 32685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:42:28.177165 32685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:42:28.178354 32685 hybrid_clock.cc:648] HybridClock initialized: now 1790577748178341 us; error 30 us; skew 500 ppm
Sep 28 06:42:28 dist-test-slave-b28r krb5kdc[32521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577748, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:42:28.181001 32685 init.cc:377] Logged in from keytab as kudu/127.18.211.124@KRBTEST.COM (short username kudu)
I20260928 06:42:28.181470 32685 webserver.cc:460] Webserver started at http://127.18.211.124:33719/ using document root <none> and password file <none>
I20260928 06:42:28.181710 32685 fs_manager.cc:359] Metadata directory not provided
I20260928 06:42:28.181761 32685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:42:28.181914 32685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:42:28.182971 32685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-2/data/instance:
uuid: "f6b8198abc874eacacee03743e61513a"
format_stamp: "Formatted at 2026-09-28 06:42:28 on dist-test-slave-b28r"
server_key: "33c0497ad71832c4940aa586ad07c0e1"
server_key_iv: "6a8695cc7a38ae65420450fb746d3685"
server_key_version: "encryptionkey@0"
I20260928 06:42:28.183332 32685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-2/wal/instance:
uuid: "f6b8198abc874eacacee03743e61513a"
format_stamp: "Formatted at 2026-09-28 06:42:28 on dist-test-slave-b28r"
server_key: "33c0497ad71832c4940aa586ad07c0e1"
server_key_iv: "6a8695cc7a38ae65420450fb746d3685"
server_key_version: "encryptionkey@0"
I20260928 06:42:28.184578 32685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:42:28.185292 32700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:28.185444 32685 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:42:28.185536 32685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-2/wal
uuid: "f6b8198abc874eacacee03743e61513a"
format_stamp: "Formatted at 2026-09-28 06:42:28 on dist-test-slave-b28r"
server_key: "33c0497ad71832c4940aa586ad07c0e1"
server_key_iv: "6a8695cc7a38ae65420450fb746d3685"
server_key_version: "encryptionkey@0"
I20260928 06:42:28.185592 32685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:42:28.201284 32678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.211.124" port: 36273 } attempt: 2
I20260928 06:42:28.201783 32681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:28.201641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:36273 (local address 127.0.0.1:40710)
0928 06:42:28.201703 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:28.201748 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.124:36273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 06:42:28.202128 32681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:28.202054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:36273 (local address 127.0.0.1:40726)
0928 06:42:28.202091 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:28.202112 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.124:36273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 06:42:28.202208 32678 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.211.124:36273: Network error: Client connection negotiation failed: client connection to 127.18.211.124:36273: connect: Connection refused (error 111)
I20260928 06:42:28.213379 32685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:42:28.215652 32685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:42:28.215778 32685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:42:28.219556 32685 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.124:36273
I20260928 06:42:28.219609 32752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.124:36273 every 8 connection(s)
I20260928 06:42:28.219937 32685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-2/data/info.pb
I20260928 06:42:28.220237 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 32685
I20260928 06:42:28.220417 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790577602522428-19277-0/minicluster-data/master-2/wal/instance
I20260928 06:42:28.220633 19277 external_mini_cluster.cc:1428] Setting key 19ea6350fd3218eebe208fac872deacb
I20260928 06:42:28.220968 32753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:42:28.223335 32753 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35959 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:42:28 dist-test-slave-b28r krb5kdc[32521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.124@KRBTEST.COM
Sep 28 06:42:28 dist-test-slave-b28r krb5kdc[32521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577748, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.124@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:42:28.231587 32605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:28.223981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33484 (local address 127.18.211.126:35959)
0928 06:42:28.224027 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:28.224029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:28.224195 (+ 166us) server_negotiation.cc:407] Connection header received
0928 06:42:28.224250 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:28.224251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:28.224310 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:28.224357 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:28.224804 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.225273 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.226141 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.226380 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.226443 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:28.228720 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:28.228721 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:28.228722 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:28.228728 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:28.230509 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:28.230781 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:28.230782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:28.230782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:28.230832 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:28.231007 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:28.231008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:28.231008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:28.231148 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:28.231216 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:28.231462 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 06:42:28.231521 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:42:28.231769 32757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:28.223934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35959 (local address 127.0.0.1:33484)
0928 06:42:28.224100 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:28.224133 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:28.224200 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:28.224392 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:28.224395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:28.224646 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:42:28.224722 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:28.224724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.225351 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.225353 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:28.225943 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:28.225945 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.226461 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.226462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:28.226535 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:28.226928 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:42:28.226938 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:42:28.228620 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:42:28.230565 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:28.230568 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:28.230569 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:28.230728 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:28.230895 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:28.230896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:28.230896 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:28.230958 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:28.231261 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:42:28.231267 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:42:28.231327 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 06:42:28.231431 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 06:42:28.231573 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":105,"threads_started":1}
I20260928 06:42:28.232048 32756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:28.221140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40740 (local address 127.18.211.124:36273)
0928 06:42:28.221334 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:28.221343 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:28.221425 (+ 82us) server_negotiation.cc:407] Connection header received
0928 06:42:28.222148 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:28.222155 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:28.222212 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:28.222367 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:28.223147 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.224320 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.225022 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.225431 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.225524 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:28.227840 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:28.227849 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:28.227850 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:28.227857 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:28.229868 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:28.230267 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:28.230270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:28.230270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:28.230380 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:28.230784 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:28.230785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:28.230785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:28.230994 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:28.231118 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:28.231898 (+ 780us) server_negotiation.cc:299] Negotiation successful
0928 06:42:28.231967 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":93,"threads_started":1}
I20260928 06:42:28.232406 32753 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.211.125" port: 34665 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:42:28 dist-test-slave-b28r krb5kdc[32521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577748, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.124@KRBTEST.COM for kudu/127.18.211.125@KRBTEST.COM
Sep 28 06:42:28 dist-test-slave-b28r krb5kdc[32521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:42:28.238716 32683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:28.232733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50570 (local address 127.18.211.125:34665)
0928 06:42:28.232783 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:28.232785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:28.232795 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:42:28.232838 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:28.232839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:28.232848 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:28.232905 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:28.233136 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.233580 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.234254 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.234464 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.234512 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:28.236419 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:28.236420 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:28.236420 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:28.236427 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:28.237676 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:28.238004 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:28.238005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:28.238006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:28.238057 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:28.238217 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:28.238218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:28.238219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:28.238354 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:28.238411 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:28.238549 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 06:42:28.238621 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:42:28.238713 32757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:28.232671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.125:34665 (local address 127.0.0.1:50570)
0928 06:42:28.232719 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:28.232730 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:28.232783 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:28.232908 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:28.232909 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:28.233047 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:42:28.233078 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:28.233079 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.233652 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.233653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:28.234168 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:28.234170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.234524 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.234525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:28.234584 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:28.234952 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:42:28.234956 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:42:28.236346 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:42:28.237752 (+ 1406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:28.237753 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:28.237753 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:28.237940 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:28.238110 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:28.238111 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:28.238111 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:28.238168 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:28.238413 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:42:28.238414 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:42:28.238459 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 06:42:28.238519 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 06:42:28.238613 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 06:42:28.239212 32753 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.211.124" port: 36273 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:42:28.239809 32605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:28.232927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33488 (local address 127.18.211.126:35959)
0928 06:42:28.232969 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:28.232971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:28.233194 (+ 223us) server_negotiation.cc:407] Connection header received
0928 06:42:28.233233 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:28.233234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:28.233243 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:28.233300 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:28.233721 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.234183 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.234945 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.235182 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.235230 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:28.237447 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:28.237448 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:28.237448 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:28.237455 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:28.238664 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:28.239030 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:28.239031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:28.239032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:28.239107 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:28.239271 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:28.239272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:28.239272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:28.239391 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:28.239567 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:28.239647 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 06:42:28.239724 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Sep 28 06:42:28 dist-test-slave-b28r krb5kdc[32521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577748, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.124@KRBTEST.COM for kudu/127.18.211.124@KRBTEST.COM
Sep 28 06:42:28 dist-test-slave-b28r krb5kdc[32521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.125@KRBTEST.COM
I20260928 06:42:28.244571 32602 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.211.124" port: 36273 } attempt: 2
I20260928 06:42:28.245441 32757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:28.239450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:36273 (local address 127.0.0.1:40752)
0928 06:42:28.239489 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:28.239497 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:28.239529 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:28.239625 (+ 96us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:28.239625 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:28.239745 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:42:28.239770 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:28.239771 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.240574 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.240575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:28.241140 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:28.241142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.241464 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.241464 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:28.241509 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:28.241835 (+ 326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:42:28.241839 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:42:28.242987 (+ 1148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:42:28.244446 (+ 1459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:28.244447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:28.244447 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:28.244669 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:28.244851 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:28.244852 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:28.244852 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:28.244904 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:28.245182 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:42:28.245182 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:42:28.245218 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 06:42:28.245276 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 06:42:28.245361 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 06:42:28.245697 32756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:28.239495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40752 (local address 127.18.211.124:36273)
0928 06:42:28.239531 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:28.239532 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:28.239539 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:42:28.239573 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:28.239573 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:28.239582 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:28.239625 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:28.239839 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.240505 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.241215 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.241409 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.241455 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:28.243059 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:28.243060 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:28.243061 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:28.243066 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:28.244368 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:28.244737 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:28.244737 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:28.244738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:28.244784 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:28.244959 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:28.244960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:28.244960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:28.245098 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:28.245159 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:28.245545 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 06:42:28.245611 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:42:28.247390 32753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6b8198abc874eacacee03743e61513a: Bootstrap starting.
I20260928 06:42:28.248045 32753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6b8198abc874eacacee03743e61513a: Neither blocks nor log segments found. Creating new log.
I20260928 06:42:28.248327 32753 log.cc:824] T 00000000000000000000000000000000 P f6b8198abc874eacacee03743e61513a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:42:28.248514 32683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:28.240387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50582 (local address 127.18.211.125:34665)
0928 06:42:28.240421 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:28.240422 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:28.240430 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:42:28.240470 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:28.240471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:28.240478 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:28.240519 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:28.240938 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.241492 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.242116 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.242358 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.242410 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:28.244651 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:28.244652 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:28.244652 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:28.244659 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:28.246253 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:28.247200 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:28.247201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:28.247202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:28.247263 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:28.247503 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:28.247504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:28.247504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:28.248094 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:28.248298 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:28.248352 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 06:42:28.248425 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Sep 28 06:42:28 dist-test-slave-b28r krb5kdc[32521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577747, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for kudu/127.18.211.124@KRBTEST.COM
I20260928 06:42:28.249450 32753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6b8198abc874eacacee03743e61513a: No bootstrap required, opened a new log
I20260928 06:42:28.250761 19277 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:42:28.251276 32753 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6b8198abc874eacacee03743e61513a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e3ca67f1d14df5b0921366d43c4b09" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35959 } } peers { permanent_uuid: "7dce2a529ac646a89a2b9423cb735bcd" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 34665 } } peers { permanent_uuid: "f6b8198abc874eacacee03743e61513a" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 36273 } }
I20260928 06:42:28.251405 32753 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6b8198abc874eacacee03743e61513a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:42:28.251438 32753 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6b8198abc874eacacee03743e61513a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b8198abc874eacacee03743e61513a, State: Initialized, Role: FOLLOWER
I20260928 06:42:28.251567 32753 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6b8198abc874eacacee03743e61513a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e3ca67f1d14df5b0921366d43c4b09" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35959 } } peers { permanent_uuid: "7dce2a529ac646a89a2b9423cb735bcd" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 34665 } } peers { permanent_uuid: "f6b8198abc874eacacee03743e61513a" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 36273 } }
I20260928 06:42:28.251711 32604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:28.244804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:36273 (local address 127.0.0.1:40766)
0928 06:42:28.245084 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:28.245096 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:28.245136 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:28.245360 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:28.245361 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:28.245506 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:42:28.245550 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:28.245552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.246332 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.246333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:28.246809 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:28.246811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.247271 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.247272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:28.247339 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:28.247665 (+ 326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:42:28.247670 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:42:28.248873 (+ 1203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:42:28.250563 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:28.250564 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:28.250565 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:28.250777 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:28.250984 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:28.250985 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:28.250985 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:28.251050 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:28.251413 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:42:28.251415 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:42:28.251463 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 06:42:28.251536 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 06:42:28.251624 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 06:42:28.251755 32758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:28.244864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40766 (local address 127.18.211.124:36273)
0928 06:42:28.245253 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:28.245254 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:28.245260 (+ 6us) server_negotiation.cc:407] Connection header received
0928 06:42:28.245276 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:28.245277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:28.245286 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:28.245348 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:28.245648 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.246255 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.246878 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.247216 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.247270 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:28.248966 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:28.248967 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:28.248968 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:28.248974 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:28.250481 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:28.250861 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:28.250862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:28.250863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:28.250925 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:28.251168 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:28.251169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:28.251169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:28.251340 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:28.251407 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:28.251566 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 06:42:28.251644 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":91,"threads_started":1}
I20260928 06:42:28.251888 32753 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6b8198abc874eacacee03743e61513a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:42:28.252108 32760 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6b8198abc874eacacee03743e61513a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e3ca67f1d14df5b0921366d43c4b09" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35959 } } peers { permanent_uuid: "7dce2a529ac646a89a2b9423cb735bcd" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 34665 } } peers { permanent_uuid: "f6b8198abc874eacacee03743e61513a" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 36273 } } }
I20260928 06:42:28.252177 32760 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6b8198abc874eacacee03743e61513a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:42:28.253254 32602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87e3ca67f1d14df5b0921366d43c4b09: Bootstrap starting.
I20260928 06:42:28.253973 32602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87e3ca67f1d14df5b0921366d43c4b09: Neither blocks nor log segments found. Creating new log.
I20260928 06:42:28.254269 32602 log.cc:824] T 00000000000000000000000000000000 P 87e3ca67f1d14df5b0921366d43c4b09: Log is configured to *not* fsync() on all Append() calls
I20260928 06:42:28.255048 32602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87e3ca67f1d14df5b0921366d43c4b09: No bootstrap required, opened a new log
I20260928 06:42:28.253942 32685 master_runner.cc:186] Error getting master registration for 127.18.211.126:35959: OK, instance_id { permanent_uuid: "87e3ca67f1d14df5b0921366d43c4b09" instance_seqno: 1790577747903660 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:42:28.256646 32685 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:42:28.257042 32602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87e3ca67f1d14df5b0921366d43c4b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e3ca67f1d14df5b0921366d43c4b09" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35959 } } peers { permanent_uuid: "7dce2a529ac646a89a2b9423cb735bcd" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 34665 } } peers { permanent_uuid: "f6b8198abc874eacacee03743e61513a" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 36273 } }
I20260928 06:42:28.257165 32602 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87e3ca67f1d14df5b0921366d43c4b09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:42:28.257195 32602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87e3ca67f1d14df5b0921366d43c4b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e3ca67f1d14df5b0921366d43c4b09, State: Initialized, Role: FOLLOWER
W20260928 06:42:28.257236 309 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f6b8198abc874eacacee03743e61513a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:42:28.257277 309 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:42:28.257477 32602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87e3ca67f1d14df5b0921366d43c4b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e3ca67f1d14df5b0921366d43c4b09" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35959 } } peers { permanent_uuid: "7dce2a529ac646a89a2b9423cb735bcd" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 34665 } } peers { permanent_uuid: "f6b8198abc874eacacee03743e61513a" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 36273 } }
I20260928 06:42:28.257798 32602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87e3ca67f1d14df5b0921366d43c4b09 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:42:28.257761 313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87e3ca67f1d14df5b0921366d43c4b09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e3ca67f1d14df5b0921366d43c4b09" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35959 } } peers { permanent_uuid: "7dce2a529ac646a89a2b9423cb735bcd" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 34665 } } peers { permanent_uuid: "f6b8198abc874eacacee03743e61513a" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 36273 } } }
I20260928 06:42:28.257952 313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87e3ca67f1d14df5b0921366d43c4b09 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:42:28.259716 32758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:28.252525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40768 (local address 127.18.211.124:36273)
0928 06:42:28.252572 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:28.252574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:28.252786 (+ 212us) server_negotiation.cc:407] Connection header received
0928 06:42:28.252868 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:28.252869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:28.252878 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:28.252919 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:28.253641 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.254220 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.255156 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.255464 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.255506 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:28.256893 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:28.256894 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:28.256894 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:28.256900 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:28.258395 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:28.258802 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:28.258803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:28.258803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:28.258874 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:28.259127 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:28.259128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:28.259129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:28.259253 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:28.259353 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:28.259559 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 06:42:28.259639 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260928 06:42:28.259944 325 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 87e3ca67f1d14df5b0921366d43c4b09: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:42:28.260058 325 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:42:28.262079 32683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:28.252244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50596 (local address 127.18.211.125:34665)
0928 06:42:28.254554 (+ 2310us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:28.254556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:28.254565 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:42:28.254588 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:28.254588 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:28.254599 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:28.254655 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:28.255648 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.256557 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.257908 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.258157 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.258191 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:28.259441 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:28.259442 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:28.259443 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:28.259450 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:28.260898 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:28.261276 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:28.261277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:28.261278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:28.261353 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:28.261550 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:28.261551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:28.261551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:28.261665 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:28.261728 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:28.261919 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 06:42:28.261993 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2284}
I20260928 06:42:28.260411 32534 master_runner.cc:186] Error getting master registration for 127.18.211.125:34665: OK, instance_id { permanent_uuid: "7dce2a529ac646a89a2b9423cb735bcd" instance_seqno: 1790577748058487 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:42:28.263357 32534 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:42:28.269037 32605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:28.254811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33496 (local address 127.18.211.126:35959)
0928 06:42:28.254845 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:28.254847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:28.258561 (+ 3714us) server_negotiation.cc:407] Connection header received
0928 06:42:28.259732 (+ 1171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:28.259733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:28.259744 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:28.259774 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:28.262947 (+ 3173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.263689 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.264597 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.264883 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.264949 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:28.266305 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:28.266306 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:28.266307 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:28.266314 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:28.267889 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:28.268269 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:28.268270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:28.268270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:28.268329 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:28.268519 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:28.268520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:28.268521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:28.268645 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:28.268747 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:28.268884 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 06:42:28.268954 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:42:28.282083 32760 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6b8198abc874eacacee03743e61513a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:42:28.282208 32760 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6b8198abc874eacacee03743e61513a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e3ca67f1d14df5b0921366d43c4b09" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35959 } } peers { permanent_uuid: "7dce2a529ac646a89a2b9423cb735bcd" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 34665 } } peers { permanent_uuid: "f6b8198abc874eacacee03743e61513a" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 36273 } }
I20260928 06:42:28.282465 32760 leader_election.cc:288] T 00000000000000000000000000000000 P f6b8198abc874eacacee03743e61513a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87e3ca67f1d14df5b0921366d43c4b09 (127.18.211.126:35959), 7dce2a529ac646a89a2b9423cb735bcd (127.18.211.125:34665)
I20260928 06:42:28.282756 32576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f6b8198abc874eacacee03743e61513a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e3ca67f1d14df5b0921366d43c4b09" is_pre_election: true
I20260928 06:42:28.282944 32576 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 87e3ca67f1d14df5b0921366d43c4b09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6b8198abc874eacacee03743e61513a in term 0.
I20260928 06:42:28.283200 32703 leader_election.cc:302] T 00000000000000000000000000000000 P f6b8198abc874eacacee03743e61513a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e3ca67f1d14df5b0921366d43c4b09, f6b8198abc874eacacee03743e61513a; no voters:
I20260928 06:42:28.283339 32760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6b8198abc874eacacee03743e61513a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:42:28.283383 32760 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6b8198abc874eacacee03743e61513a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:42:28.283404 32760 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6b8198abc874eacacee03743e61513a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:42:28.284322 32760 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6b8198abc874eacacee03743e61513a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e3ca67f1d14df5b0921366d43c4b09" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35959 } } peers { permanent_uuid: "7dce2a529ac646a89a2b9423cb735bcd" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 34665 } } peers { permanent_uuid: "f6b8198abc874eacacee03743e61513a" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 36273 } }
I20260928 06:42:28.284449 32760 leader_election.cc:288] T 00000000000000000000000000000000 P f6b8198abc874eacacee03743e61513a [CANDIDATE]: Term 1 election: Requested vote from peers 87e3ca67f1d14df5b0921366d43c4b09 (127.18.211.126:35959), 7dce2a529ac646a89a2b9423cb735bcd (127.18.211.125:34665)
I20260928 06:42:28.284636 32576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f6b8198abc874eacacee03743e61513a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e3ca67f1d14df5b0921366d43c4b09"
I20260928 06:42:28.284727 32576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87e3ca67f1d14df5b0921366d43c4b09 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:42:28.285633 32576 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 87e3ca67f1d14df5b0921366d43c4b09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6b8198abc874eacacee03743e61513a in term 1.
I20260928 06:42:28.285820 32703 leader_election.cc:302] T 00000000000000000000000000000000 P f6b8198abc874eacacee03743e61513a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e3ca67f1d14df5b0921366d43c4b09, f6b8198abc874eacacee03743e61513a; no voters:
I20260928 06:42:28.285916 32760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6b8198abc874eacacee03743e61513a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:42:28.286072 32760 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6b8198abc874eacacee03743e61513a [term 1 LEADER]: Becoming Leader. State: Replica: f6b8198abc874eacacee03743e61513a, State: Running, Role: LEADER
I20260928 06:42:28.286164 32760 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6b8198abc874eacacee03743e61513a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e3ca67f1d14df5b0921366d43c4b09" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35959 } } peers { permanent_uuid: "7dce2a529ac646a89a2b9423cb735bcd" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 34665 } } peers { permanent_uuid: "f6b8198abc874eacacee03743e61513a" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 36273 } }
I20260928 06:42:28.286548 32760 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6b8198abc874eacacee03743e61513a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6b8198abc874eacacee03743e61513a. Latest consensus state: current_term: 1 leader_uuid: "f6b8198abc874eacacee03743e61513a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e3ca67f1d14df5b0921366d43c4b09" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35959 } } peers { permanent_uuid: "7dce2a529ac646a89a2b9423cb735bcd" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 34665 } } peers { permanent_uuid: "f6b8198abc874eacacee03743e61513a" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 36273 } } }
I20260928 06:42:28.286626 32760 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6b8198abc874eacacee03743e61513a [sys.catalog]: This master's current role is: LEADER
I20260928 06:42:28.286819 329 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:42:28.287108 329 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:42:28.288868 32576 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 87e3ca67f1d14df5b0921366d43c4b09 [term 1 FOLLOWER]: Refusing update from remote peer f6b8198abc874eacacee03743e61513a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:42:28.289531 327 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f6b8198abc874eacacee03743e61513a [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e3ca67f1d14df5b0921366d43c4b09" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:42:28.291059 313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87e3ca67f1d14df5b0921366d43c4b09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6b8198abc874eacacee03743e61513a. Latest consensus state: current_term: 1 leader_uuid: "f6b8198abc874eacacee03743e61513a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e3ca67f1d14df5b0921366d43c4b09" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35959 } } peers { permanent_uuid: "7dce2a529ac646a89a2b9423cb735bcd" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 34665 } } peers { permanent_uuid: "f6b8198abc874eacacee03743e61513a" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 36273 } } }
I20260928 06:42:28.291195 313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87e3ca67f1d14df5b0921366d43c4b09 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:42:28.292636 327 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6b8198abc874eacacee03743e61513a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f6b8198abc874eacacee03743e61513a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e3ca67f1d14df5b0921366d43c4b09" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35959 } } peers { permanent_uuid: "7dce2a529ac646a89a2b9423cb735bcd" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 34665 } } peers { permanent_uuid: "f6b8198abc874eacacee03743e61513a" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 36273 } } }
I20260928 06:42:28.292724 327 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6b8198abc874eacacee03743e61513a [sys.catalog]: This master's current role is: LEADER
I20260928 06:42:28.293241 313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87e3ca67f1d14df5b0921366d43c4b09 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f6b8198abc874eacacee03743e61513a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e3ca67f1d14df5b0921366d43c4b09" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35959 } } peers { permanent_uuid: "7dce2a529ac646a89a2b9423cb735bcd" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 34665 } } peers { permanent_uuid: "f6b8198abc874eacacee03743e61513a" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 36273 } } }
I20260928 06:42:28.293316 313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87e3ca67f1d14df5b0921366d43c4b09 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:42:28.293319 32678 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.211.124" port: 36273 } attempt: 3
I20260928 06:42:28.293682 330 mvcc.cc:205] Tried to move back new op lower bound from 7334206456988262400 to 7334206456980762624. Current Snapshot: MvccSnapshot[applied={T|T < 7334206456988262400}]
Sep 28 06:42:28 dist-test-slave-b28r krb5kdc[32521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577748, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.125@KRBTEST.COM for kudu/127.18.211.124@KRBTEST.COM
I20260928 06:42:28.301608 32681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:28.293609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:36273 (local address 127.0.0.1:40776)
0928 06:42:28.293692 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:28.293703 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:28.293740 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:28.294062 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:28.294063 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:28.294243 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:42:28.294297 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:28.294299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.295247 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.295249 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:28.296053 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:28.296056 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.296523 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.296524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:28.296599 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:28.297092 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:42:28.297112 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:42:28.298659 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:42:28.300487 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:28.300489 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:28.300489 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:28.300707 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:28.300938 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:28.300939 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:28.300939 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:28.301007 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:28.301285 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:42:28.301286 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:42:28.301343 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 06:42:28.301405 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 06:42:28.301504 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 06:42:28.301939 32758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:28.293888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40776 (local address 127.18.211.124:36273)
0928 06:42:28.293945 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:28.293947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:28.293954 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:42:28.293973 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:28.293974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:28.293984 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:28.294034 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:28.294374 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.295165 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.296135 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:28.296445 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:28.296514 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:28.298748 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:28.298750 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:28.298750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:28.298757 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:28.300398 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:28.300797 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:28.300798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:28.300798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:28.300875 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:28.301060 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:28.301061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:28.301076 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:28.301226 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:28.301291 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:28.301772 (+ 481us) server_negotiation.cc:299] Negotiation successful
0928 06:42:28.301849 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:42:28.303191 329 catalog_manager.cc:1347] Generated new cluster ID: ac5b78c3ad2246728da364491ff0b51e
I20260928 06:42:28.303254 329 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:42:28.303560 32678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dce2a529ac646a89a2b9423cb735bcd: Bootstrap starting.
I20260928 06:42:28.304296 32678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dce2a529ac646a89a2b9423cb735bcd: Neither blocks nor log segments found. Creating new log.
I20260928 06:42:28.304622 32678 log.cc:824] T 00000000000000000000000000000000 P 7dce2a529ac646a89a2b9423cb735bcd: Log is configured to *not* fsync() on all Append() calls
I20260928 06:42:28.305799 32678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dce2a529ac646a89a2b9423cb735bcd: No bootstrap required, opened a new log
I20260928 06:42:28.307844 32678 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dce2a529ac646a89a2b9423cb735bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e3ca67f1d14df5b0921366d43c4b09" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35959 } } peers { permanent_uuid: "7dce2a529ac646a89a2b9423cb735bcd" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 34665 } } peers { permanent_uuid: "f6b8198abc874eacacee03743e61513a" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 36273 } }
I20260928 06:42:28.307999 32678 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7dce2a529ac646a89a2b9423cb735bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:42:28.308040 32678 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dce2a529ac646a89a2b9423cb735bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dce2a529ac646a89a2b9423cb735bcd, State: Initialized, Role: FOLLOWER
I20260928 06:42:28.308157 32678 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dce2a529ac646a89a2b9423cb735bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e3ca67f1d14df5b0921366d43c4b09" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35959 } } peers { permanent_uuid: "7dce2a529ac646a89a2b9423cb735bcd" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 34665 } } peers { permanent_uuid: "f6b8198abc874eacacee03743e61513a" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 36273 } }
I20260928 06:42:28.308483 32678 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dce2a529ac646a89a2b9423cb735bcd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:42:28.308724 32650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dce2a529ac646a89a2b9423cb735bcd [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:42:28.308727 32651 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f6b8198abc874eacacee03743e61513a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7dce2a529ac646a89a2b9423cb735bcd"
I20260928 06:42:28.309168 32652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f6b8198abc874eacacee03743e61513a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7dce2a529ac646a89a2b9423cb735bcd" is_pre_election: true
I20260928 06:42:28.309396 336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dce2a529ac646a89a2b9423cb735bcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e3ca67f1d14df5b0921366d43c4b09" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35959 } } peers { permanent_uuid: "7dce2a529ac646a89a2b9423cb735bcd" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 34665 } } peers { permanent_uuid: "f6b8198abc874eacacee03743e61513a" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 36273 } } }
I20260928 06:42:28.309506 336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dce2a529ac646a89a2b9423cb735bcd [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:42:28.309703 32650 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7dce2a529ac646a89a2b9423cb735bcd [term 1 FOLLOWER]: Refusing update from remote peer f6b8198abc874eacacee03743e61513a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:42:28.309933 32760 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f6b8198abc874eacacee03743e61513a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7dce2a529ac646a89a2b9423cb735bcd" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 34665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:42:28.311569 336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dce2a529ac646a89a2b9423cb735bcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6b8198abc874eacacee03743e61513a. Latest consensus state: current_term: 1 leader_uuid: "f6b8198abc874eacacee03743e61513a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e3ca67f1d14df5b0921366d43c4b09" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35959 } } peers { permanent_uuid: "7dce2a529ac646a89a2b9423cb735bcd" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 34665 } } peers { permanent_uuid: "f6b8198abc874eacacee03743e61513a" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 36273 } } }
I20260928 06:42:28.312050 336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dce2a529ac646a89a2b9423cb735bcd [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:42:28.312120 341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dce2a529ac646a89a2b9423cb735bcd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f6b8198abc874eacacee03743e61513a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e3ca67f1d14df5b0921366d43c4b09" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35959 } } peers { permanent_uuid: "7dce2a529ac646a89a2b9423cb735bcd" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 34665 } } peers { permanent_uuid: "f6b8198abc874eacacee03743e61513a" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 36273 } } }
I20260928 06:42:28.312176 341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dce2a529ac646a89a2b9423cb735bcd [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:42:28.313536 351 catalog_manager.cc:1259] Loaded cluster ID: ac5b78c3ad2246728da364491ff0b51e
I20260928 06:42:28.313617 351 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7dce2a529ac646a89a2b9423cb735bcd: loading cluster ID for follower catalog manager: success
W20260928 06:42:28.313738 351 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 7dce2a529ac646a89a2b9423cb735bcd: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260928 06:42:28.313802 351 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:42:28.317292 327 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6b8198abc874eacacee03743e61513a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f6b8198abc874eacacee03743e61513a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e3ca67f1d14df5b0921366d43c4b09" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35959 } } peers { permanent_uuid: "7dce2a529ac646a89a2b9423cb735bcd" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 34665 } } peers { permanent_uuid: "f6b8198abc874eacacee03743e61513a" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 36273 } } }
I20260928 06:42:28.317400 327 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6b8198abc874eacacee03743e61513a [sys.catalog]: This master's current role is: LEADER
I20260928 06:42:28.318265 329 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:42:28.318953 329 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:42:28.330081 329 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6b8198abc874eacacee03743e61513a: Generated new TSK 0
I20260928 06:42:28.330292 329 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:42:29.260757 325 catalog_manager.cc:1259] Loaded cluster ID: ac5b78c3ad2246728da364491ff0b51e
I20260928 06:42:29.260838 325 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 87e3ca67f1d14df5b0921366d43c4b09: loading cluster ID for follower catalog manager: success
I20260928 06:42:29.261668 325 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 87e3ca67f1d14df5b0921366d43c4b09: acquiring CA information for follower catalog manager: success
I20260928 06:42:29.261976 325 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 87e3ca67f1d14df5b0921366d43c4b09: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:42:29.315026 351 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7dce2a529ac646a89a2b9423cb735bcd: acquiring CA information for follower catalog manager: success
I20260928 06:42:29.315439 351 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7dce2a529ac646a89a2b9423cb735bcd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:42:29.389796 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 32534
I20260928 06:42:29.394976 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 32609
I20260928 06:42:29.399260 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 32685
2026-09-28T06:42:29Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2666 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:42:29 dist-test-slave-b28r krb5kdc[360](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:42:29 dist-test-slave-b28r krb5kdc[360](info): set up 2 sockets
Sep 28 06:42:29 dist-test-slave-b28r krb5kdc[360](info): commencing operation
krb5kdc: starting...
W20260928 06:42:30.359918 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:42:30 dist-test-slave-b28r krb5kdc[360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:42:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:42:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.211.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
I20260928 06:42:30.468132 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:32947
--webserver_interface=127.18.211.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:41723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:32947
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:42:30.546653 373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:42:30.546864 373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:42:30.546896 373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:42:30.548072 373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:42:30.548115 373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:42:30.548132 373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:42:30.548148 373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:42:30.548162 373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:42:30.549613 373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:41723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:32947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:32947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:42:30.549853 373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:42:30.550055 373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:42:30.554593 378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:30.554589 379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:30.554601 381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:42:30.554867 373 server_base.cc:1034] running on GCE node
I20260928 06:42:30.555111 373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:42:30.555411 373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:42:30.556555 373 hybrid_clock.cc:648] HybridClock initialized: now 1790577750556534 us; error 38 us; skew 500 ppm
Sep 28 06:42:30 dist-test-slave-b28r krb5kdc[360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577750, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:42:30.558897 373 init.cc:377] Logged in from keytab as kudu/127.18.211.126@KRBTEST.COM (short username kudu)
I20260928 06:42:30.559373 373 webserver.cc:460] Webserver started at http://127.18.211.126:37949/ using document root <none> and password file <none>
I20260928 06:42:30.559571 373 fs_manager.cc:359] Metadata directory not provided
I20260928 06:42:30.559618 373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:42:30.559741 373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:42:30.560724 373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "bf0ccbee4ba4468b88aaee622dd9daa4"
format_stamp: "Formatted at 2026-09-28 06:42:30 on dist-test-slave-b28r"
server_key: "f0a4f7b89ff5c8b67acbac9c7231a02a"
server_key_iv: "011d588fbf5106c699e18463a58754a9"
server_key_version: "encryptionkey@0"
I20260928 06:42:30.561045 373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "bf0ccbee4ba4468b88aaee622dd9daa4"
format_stamp: "Formatted at 2026-09-28 06:42:30 on dist-test-slave-b28r"
server_key: "f0a4f7b89ff5c8b67acbac9c7231a02a"
server_key_iv: "011d588fbf5106c699e18463a58754a9"
server_key_version: "encryptionkey@0"
I20260928 06:42:30.562350 373 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:42:30.563005 388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:30.563158 373 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:42:30.563272 373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "bf0ccbee4ba4468b88aaee622dd9daa4"
format_stamp: "Formatted at 2026-09-28 06:42:30 on dist-test-slave-b28r"
server_key: "f0a4f7b89ff5c8b67acbac9c7231a02a"
server_key_iv: "011d588fbf5106c699e18463a58754a9"
server_key_version: "encryptionkey@0"
I20260928 06:42:30.563330 373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:42:30.576552 373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:42:30.578392 373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:42:30.578526 373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:42:30.582345 373 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:32947
I20260928 06:42:30.582379 440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:32947 every 8 connection(s)
I20260928 06:42:30.582707 373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:42:30.583565 441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:42:30.585753 441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf0ccbee4ba4468b88aaee622dd9daa4: Bootstrap starting.
I20260928 06:42:30.586300 441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf0ccbee4ba4468b88aaee622dd9daa4: Neither blocks nor log segments found. Creating new log.
I20260928 06:42:30.586548 441 log.cc:824] T 00000000000000000000000000000000 P bf0ccbee4ba4468b88aaee622dd9daa4: Log is configured to *not* fsync() on all Append() calls
I20260928 06:42:30.587198 441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf0ccbee4ba4468b88aaee622dd9daa4: No bootstrap required, opened a new log
I20260928 06:42:30.588794 441 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf0ccbee4ba4468b88aaee622dd9daa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0ccbee4ba4468b88aaee622dd9daa4" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 32947 } }
I20260928 06:42:30.588929 441 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf0ccbee4ba4468b88aaee622dd9daa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:42:30.588958 441 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf0ccbee4ba4468b88aaee622dd9daa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf0ccbee4ba4468b88aaee622dd9daa4, State: Initialized, Role: FOLLOWER
I20260928 06:42:30.589085 441 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf0ccbee4ba4468b88aaee622dd9daa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0ccbee4ba4468b88aaee622dd9daa4" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 32947 } }
I20260928 06:42:30.589148 441 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf0ccbee4ba4468b88aaee622dd9daa4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:42:30.589169 441 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf0ccbee4ba4468b88aaee622dd9daa4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:42:30.589195 441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf0ccbee4ba4468b88aaee622dd9daa4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:42:30.589677 441 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf0ccbee4ba4468b88aaee622dd9daa4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0ccbee4ba4468b88aaee622dd9daa4" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 32947 } }
I20260928 06:42:30.589764 441 leader_election.cc:302] T 00000000000000000000000000000000 P bf0ccbee4ba4468b88aaee622dd9daa4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf0ccbee4ba4468b88aaee622dd9daa4; no voters:
I20260928 06:42:30.589916 441 leader_election.cc:288] T 00000000000000000000000000000000 P bf0ccbee4ba4468b88aaee622dd9daa4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:42:30.589958 444 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf0ccbee4ba4468b88aaee622dd9daa4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:42:30.590106 444 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf0ccbee4ba4468b88aaee622dd9daa4 [term 1 LEADER]: Becoming Leader. State: Replica: bf0ccbee4ba4468b88aaee622dd9daa4, State: Running, Role: LEADER
I20260928 06:42:30.590152 441 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf0ccbee4ba4468b88aaee622dd9daa4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:42:30.590198 444 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf0ccbee4ba4468b88aaee622dd9daa4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0ccbee4ba4468b88aaee622dd9daa4" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 32947 } }
I20260928 06:42:30.590549 446 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf0ccbee4ba4468b88aaee622dd9daa4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf0ccbee4ba4468b88aaee622dd9daa4. Latest consensus state: current_term: 1 leader_uuid: "bf0ccbee4ba4468b88aaee622dd9daa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0ccbee4ba4468b88aaee622dd9daa4" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 32947 } } }
I20260928 06:42:30.590632 446 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf0ccbee4ba4468b88aaee622dd9daa4 [sys.catalog]: This master's current role is: LEADER
I20260928 06:42:30.590667 445 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf0ccbee4ba4468b88aaee622dd9daa4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf0ccbee4ba4468b88aaee622dd9daa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf0ccbee4ba4468b88aaee622dd9daa4" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 32947 } } }
I20260928 06:42:30.590728 445 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf0ccbee4ba4468b88aaee622dd9daa4 [sys.catalog]: This master's current role is: LEADER
I20260928 06:42:30.590997 453 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:42:30.591547 453 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:42:30.592592 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 373
I20260928 06:42:30.592702 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:42:30.592908 19277 external_mini_cluster.cc:1428] Setting key da8edd92b5dfe29c50e186b6581b8a00
I20260928 06:42:30.593025 453 catalog_manager.cc:1347] Generated new cluster ID: 24e974162b6642f7b119a5ecc1120b65
I20260928 06:42:30.593067 453 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:42:30 dist-test-slave-b28r krb5kdc[360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:42:30.605808 465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:30.593457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60970 (local address 127.18.211.126:32947)
0928 06:42:30.595289 (+ 1832us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:30.595297 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:30.595332 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:42:30.595442 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:30.595449 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:30.595514 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:30.595656 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:30.596369 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:30.597838 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:30.598849 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:30.599524 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:30.599633 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:30.602321 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:30.602325 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:30.602325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:30.602338 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:30.604163 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:30.604549 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:30.604552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:30.604553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:30.604650 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:30.604832 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:30.604833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:30.604833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:30.605103 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:30.605211 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:30.605339 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 06:42:30.605560 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1782,"thread_start_us":61,"threads_started":1}
I20260928 06:42:30.606701 19277 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:42:30.607811 453 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:42:30.608278 453 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:42:30.615837 453 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf0ccbee4ba4468b88aaee622dd9daa4: Generated new TSK 0
I20260928 06:42:30.616011 453 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:42:30.660934 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 373
2026-09-28T06:42:30Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1260 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:42:30 dist-test-slave-b28r krb5kdc[469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:42:30 dist-test-slave-b28r krb5kdc[469](info): set up 2 sockets
Sep 28 06:42:30 dist-test-slave-b28r krb5kdc[469](info): commencing operation
krb5kdc: starting...
W20260928 06:42:31.611997 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:42:31 dist-test-slave-b28r krb5kdc[469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:42:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:42:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.211.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
I20260928 06:42:31.719550 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:33269
--webserver_interface=127.18.211.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:33269
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:42:31.798138 482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:42:31.798357 482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:42:31.798388 482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:42:31.799562 482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:42:31.799602 482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:42:31.799619 482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:42:31.799635 482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:42:31.799651 482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:42:31.801076 482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:33269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:33269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:42:31.801314 482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:42:31.801512 482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:42:31.806143 487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:31.806149 490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:31.806170 488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:42:31.806573 482 server_base.cc:1034] running on GCE node
I20260928 06:42:31.806797 482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:42:31.807049 482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:42:31.808202 482 hybrid_clock.cc:648] HybridClock initialized: now 1790577751808188 us; error 29 us; skew 500 ppm
Sep 28 06:42:31 dist-test-slave-b28r krb5kdc[469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577751, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:42:31.810580 482 init.cc:377] Logged in from keytab as kudu/127.18.211.126@KRBTEST.COM (short username kudu)
I20260928 06:42:31.810981 482 webserver.cc:460] Webserver started at http://127.18.211.126:34409/ using document root <none> and password file <none>
I20260928 06:42:31.811185 482 fs_manager.cc:359] Metadata directory not provided
I20260928 06:42:31.811233 482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:42:31.811328 482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:42:31.812350 482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "98ea104104c0443bbfad48f3cd5f272e"
format_stamp: "Formatted at 2026-09-28 06:42:31 on dist-test-slave-b28r"
server_key: "56564d69f13ab234aa9b81c472c2b297"
server_key_iv: "9f14f35f2f6196e399d4e389b3273995"
server_key_version: "encryptionkey@0"
I20260928 06:42:31.812651 482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "98ea104104c0443bbfad48f3cd5f272e"
format_stamp: "Formatted at 2026-09-28 06:42:31 on dist-test-slave-b28r"
server_key: "56564d69f13ab234aa9b81c472c2b297"
server_key_iv: "9f14f35f2f6196e399d4e389b3273995"
server_key_version: "encryptionkey@0"
I20260928 06:42:31.813830 482 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:42:31.814455 497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:31.814589 482 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:31.814666 482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "98ea104104c0443bbfad48f3cd5f272e"
format_stamp: "Formatted at 2026-09-28 06:42:31 on dist-test-slave-b28r"
server_key: "56564d69f13ab234aa9b81c472c2b297"
server_key_iv: "9f14f35f2f6196e399d4e389b3273995"
server_key_version: "encryptionkey@0"
I20260928 06:42:31.814736 482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:42:31.842739 482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:42:31.845033 482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:42:31.845165 482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:42:31.849068 482 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:33269
I20260928 06:42:31.849124 549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:33269 every 8 connection(s)
I20260928 06:42:31.849449 482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:42:31.850422 550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:42:31.852747 550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e: Bootstrap starting.
I20260928 06:42:31.853319 550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e: Neither blocks nor log segments found. Creating new log.
I20260928 06:42:31.853574 550 log.cc:824] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e: Log is configured to *not* fsync() on all Append() calls
I20260928 06:42:31.854004 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 482
I20260928 06:42:31.854127 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:42:31.854329 19277 external_mini_cluster.cc:1428] Setting key 7c7c6743db10981e80b1abee58e898bd
I20260928 06:42:31.854331 550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e: No bootstrap required, opened a new log
I20260928 06:42:31.856598 550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ea104104c0443bbfad48f3cd5f272e" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 33269 } }
I20260928 06:42:31.856739 550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:42:31.856760 550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98ea104104c0443bbfad48f3cd5f272e, State: Initialized, Role: FOLLOWER
I20260928 06:42:31.856870 550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ea104104c0443bbfad48f3cd5f272e" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 33269 } }
I20260928 06:42:31.856937 550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:42:31.856957 550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:42:31.857002 550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:42:31.857537 550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ea104104c0443bbfad48f3cd5f272e" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 33269 } }
I20260928 06:42:31.857622 550 leader_election.cc:302] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98ea104104c0443bbfad48f3cd5f272e; no voters:
I20260928 06:42:31.857770 550 leader_election.cc:288] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:42:31.857813 555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:42:31.857972 555 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [term 1 LEADER]: Becoming Leader. State: Replica: 98ea104104c0443bbfad48f3cd5f272e, State: Running, Role: LEADER
I20260928 06:42:31.858022 550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:42:31.858086 555 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ea104104c0443bbfad48f3cd5f272e" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 33269 } }
I20260928 06:42:31.858413 555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98ea104104c0443bbfad48f3cd5f272e. Latest consensus state: current_term: 1 leader_uuid: "98ea104104c0443bbfad48f3cd5f272e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ea104104c0443bbfad48f3cd5f272e" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 33269 } } }
I20260928 06:42:31.858628 555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [sys.catalog]: This master's current role is: LEADER
I20260928 06:42:31.858565 556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98ea104104c0443bbfad48f3cd5f272e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ea104104c0443bbfad48f3cd5f272e" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 33269 } } }
I20260928 06:42:31.859010 556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [sys.catalog]: This master's current role is: LEADER
W20260928 06:42:31.859804 571 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:42:31.859858 571 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:42:31.859906 564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:42:31.860061 564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 06:42:31 dist-test-slave-b28r krb5kdc[469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:42:31.861657 564 catalog_manager.cc:1347] Generated new cluster ID: 548c21bb1d884ab1bacc55b42743b6d3
I20260928 06:42:31.861708 564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:42:31.865464 554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:31.854825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58020 (local address 127.18.211.126:33269)
0928 06:42:31.855010 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:31.855018 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:31.855044 (+ 26us) server_negotiation.cc:407] Connection header received
0928 06:42:31.855650 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:31.855657 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:31.855708 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:31.855837 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:31.856549 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:31.857833 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:31.858854 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:31.859399 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:31.859560 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:31.861869 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:31.861873 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:31.861874 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:31.861885 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:31.863914 (+ 2029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:31.864281 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:31.864284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:31.864285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:31.864393 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:31.864566 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:31.864567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:31.864567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:31.864887 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:31.865032 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:31.865112 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 06:42:31.865250 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":87,"threads_started":1}
I20260928 06:42:31.869463 564 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:42:31.870151 564 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:42:31.873903 564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e: Generated new TSK 0
I20260928 06:42:31.874060 564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.211.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab.
Entry for principal kudu/127.18.211.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab.
I20260928 06:42:31.903465 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.65:0
--local_ip_for_outbound_sockets=127.18.211.65
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:33269
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:42:31.983773 576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:42:31.983964 576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:42:31.983989 576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:42:31.985140 576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:42:31.985262 576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:42:31.985288 576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.65
I20260928 06:42:31.986718 576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:33269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.65
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:42:31.986922 576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:42:31.987169 576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:42:31.991846 576 server_base.cc:1034] running on GCE node
W20260928 06:42:31.991806 581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:31.991865 582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:31.991815 584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:42:31.992262 576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:42:31.992462 576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:42:31.993602 576 hybrid_clock.cc:648] HybridClock initialized: now 1790577751993588 us; error 28 us; skew 500 ppm
Sep 28 06:42:31 dist-test-slave-b28r krb5kdc[469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577751, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:42:31.996068 576 init.cc:377] Logged in from keytab as kudu/127.18.211.65@KRBTEST.COM (short username kudu)
I20260928 06:42:31.996440 576 webserver.cc:460] Webserver started at http://127.18.211.65:46339/ using document root <none> and password file <none>
I20260928 06:42:31.996623 576 fs_manager.cc:359] Metadata directory not provided
I20260928 06:42:31.996671 576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:42:31.996764 576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:42:31.997746 576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/data/instance:
uuid: "a8a75496766348418bca6d243610c5dc"
format_stamp: "Formatted at 2026-09-28 06:42:31 on dist-test-slave-b28r"
server_key: "2938121db54c2b6505cee2d6a936f3d6"
server_key_iv: "0909d0eaf6362a26fda324f17f0aa3ce"
server_key_version: "encryptionkey@0"
I20260928 06:42:31.998024 576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance:
uuid: "a8a75496766348418bca6d243610c5dc"
format_stamp: "Formatted at 2026-09-28 06:42:31 on dist-test-slave-b28r"
server_key: "2938121db54c2b6505cee2d6a936f3d6"
server_key_iv: "0909d0eaf6362a26fda324f17f0aa3ce"
server_key_version: "encryptionkey@0"
I20260928 06:42:31.999181 576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:42:31.999806 591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:31.999949 576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:42:32.000021 576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/wal
uuid: "a8a75496766348418bca6d243610c5dc"
format_stamp: "Formatted at 2026-09-28 06:42:31 on dist-test-slave-b28r"
server_key: "2938121db54c2b6505cee2d6a936f3d6"
server_key_iv: "0909d0eaf6362a26fda324f17f0aa3ce"
server_key_version: "encryptionkey@0"
I20260928 06:42:32.000082 576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:42:32.009912 576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:42:32.011960 576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:42:32.012086 576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:42:32.012291 576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:42:32.012568 576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:42:32.012600 576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:32.012627 576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:42:32.012657 576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:32.018270 576 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.65:42529
I20260928 06:42:32.018319 705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.65:42529 every 8 connection(s)
I20260928 06:42:32.018640 576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:42:32 dist-test-slave-b28r krb5kdc[469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577751, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:42:32.028041 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 576
I20260928 06:42:32.028131 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance
I20260928 06:42:32.028337 19277 external_mini_cluster.cc:1428] Setting key 031238379f66014f2fe4c8fc831cd9fc
I20260928 06:42:32.028789 554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:32.019339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:36957 (local address 127.18.211.126:33269)
0928 06:42:32.019382 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:32.019384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:32.019572 (+ 188us) server_negotiation.cc:407] Connection header received
0928 06:42:32.020077 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:32.020078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:32.020088 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:32.020138 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:32.021055 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.021732 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.023227 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.023547 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.023615 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:32.026014 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:32.026015 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:32.026016 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:32.026023 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:32.027306 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:32.027674 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:32.027675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:32.027676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:32.027727 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:32.028009 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:32.028010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:32.028011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:32.028164 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:32.028240 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:32.028595 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 06:42:32.028696 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:42:32.029090 709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:32.019319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.65:36957)
0928 06:42:32.019458 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:32.019493 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:32.019979 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:32.020194 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:32.020198 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:32.020725 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:42:32.020947 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:32.020953 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.021841 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.021842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:32.023148 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:32.023151 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.023628 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.023629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:32.023781 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:32.024352 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:42:32.024362 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:42:32.025921 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:42:32.027385 (+ 1464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:32.027392 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:32.027394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:32.027602 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:32.027781 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:32.027782 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:32.027782 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:32.027928 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:32.028232 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:42:32.028240 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:42:32.028443 (+ 203us) client_negotiation.cc:769] Sending connection context
0928 06:42:32.028556 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 06:42:32.028716 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
I20260928 06:42:32.030979 706 heartbeater.cc:347] Connected to a master server at 127.18.211.126:33269
I20260928 06:42:32.031106 706 heartbeater.cc:464] Registering TS with master...
I20260928 06:42:32.031324 706 heartbeater.cc:511] Master 127.18.211.126:33269 requested a full tablet report, sending...
I20260928 06:42:32.031898 514 ts_manager.cc:194] Registered new tserver with Master: a8a75496766348418bca6d243610c5dc (127.18.211.65:42529)
I20260928 06:42:32.032523 514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.65@KRBTEST.COM'} at 127.18.211.65:36957
WARNING: no policy specified for kudu/127.18.211.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab.
Entry for principal kudu/127.18.211.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab.
I20260928 06:42:32.049983 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.66:0
--local_ip_for_outbound_sockets=127.18.211.66
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:33269
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:42:32.130105 712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:42:32.130314 712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:42:32.130347 712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:42:32.131528 712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:42:32.131570 712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:42:32.131605 712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.66
I20260928 06:42:32.133076 712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:33269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.66
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:42:32.133308 712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:42:32.133520 712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:42:32.138236 717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:32.138285 718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:32.138253 720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:42:32.138526 712 server_base.cc:1034] running on GCE node
I20260928 06:42:32.138722 712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:42:32.138969 712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:42:32.140105 712 hybrid_clock.cc:648] HybridClock initialized: now 1790577752140089 us; error 30 us; skew 500 ppm
Sep 28 06:42:32 dist-test-slave-b28r krb5kdc[469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577752, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:42:32.142499 712 init.cc:377] Logged in from keytab as kudu/127.18.211.66@KRBTEST.COM (short username kudu)
I20260928 06:42:32.142925 712 webserver.cc:460] Webserver started at http://127.18.211.66:38643/ using document root <none> and password file <none>
I20260928 06:42:32.143149 712 fs_manager.cc:359] Metadata directory not provided
I20260928 06:42:32.143203 712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:42:32.143325 712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:42:32.144368 712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/data/instance:
uuid: "a34c9d02780440f68a330a63df8da0a9"
format_stamp: "Formatted at 2026-09-28 06:42:32 on dist-test-slave-b28r"
server_key: "bbf315a9465a08108bd594d407748d18"
server_key_iv: "929f02d6a143257417e06b4f4b5e5f14"
server_key_version: "encryptionkey@0"
I20260928 06:42:32.144706 712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance:
uuid: "a34c9d02780440f68a330a63df8da0a9"
format_stamp: "Formatted at 2026-09-28 06:42:32 on dist-test-slave-b28r"
server_key: "bbf315a9465a08108bd594d407748d18"
server_key_iv: "929f02d6a143257417e06b4f4b5e5f14"
server_key_version: "encryptionkey@0"
I20260928 06:42:32.145933 712 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:42:32.146688 727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:32.146855 712 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:42:32.146939 712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/wal
uuid: "a34c9d02780440f68a330a63df8da0a9"
format_stamp: "Formatted at 2026-09-28 06:42:32 on dist-test-slave-b28r"
server_key: "bbf315a9465a08108bd594d407748d18"
server_key_iv: "929f02d6a143257417e06b4f4b5e5f14"
server_key_version: "encryptionkey@0"
I20260928 06:42:32.147001 712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:42:32.161278 712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:42:32.163690 712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:42:32.163831 712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:42:32.164063 712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:42:32.164381 712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:42:32.164414 712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:32.164443 712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:42:32.164466 712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:32.170140 712 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.66:36929
I20260928 06:42:32.170204 841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.66:36929 every 8 connection(s)
I20260928 06:42:32.170501 712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
I20260928 06:42:32.174168 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 712
I20260928 06:42:32.174271 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance
I20260928 06:42:32.174469 19277 external_mini_cluster.cc:1428] Setting key 91d93f836c70223aa1ffbefe2d5ea732
Sep 28 06:42:32 dist-test-slave-b28r krb5kdc[469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577752, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:42:32.179737 554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:32.171192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:55893 (local address 127.18.211.126:33269)
0928 06:42:32.171238 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:32.171240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:32.171461 (+ 221us) server_negotiation.cc:407] Connection header received
0928 06:42:32.171940 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:32.171940 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:32.171950 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:32.171997 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:32.172904 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.173411 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.174606 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.174849 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.174909 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:32.177130 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:32.177132 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:32.177132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:32.177140 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:32.178339 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:32.178779 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:32.178780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:32.178780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:32.178834 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:32.179058 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:32.179059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:32.179060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:32.179198 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:32.179256 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:32.179582 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 06:42:32.179652 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:42:32.179968 844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:32.171185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.66:55893)
0928 06:42:32.171337 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:32.171375 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:32.171865 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:32.172057 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:32.172060 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:32.172600 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:42:32.172802 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:32.172806 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.173517 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.173518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:32.174508 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:32.174512 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.174928 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.174930 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:32.175072 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:32.175641 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:42:32.175651 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:42:32.177041 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:42:32.178416 (+ 1375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:32.178424 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:32.178426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:32.178709 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:32.178889 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:32.178890 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:32.178890 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:32.179001 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:32.179255 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:42:32.179263 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:42:32.179430 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 06:42:32.179550 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 06:42:32.179731 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
I20260928 06:42:32.180485 842 heartbeater.cc:347] Connected to a master server at 127.18.211.126:33269
I20260928 06:42:32.180580 842 heartbeater.cc:464] Registering TS with master...
I20260928 06:42:32.180756 842 heartbeater.cc:511] Master 127.18.211.126:33269 requested a full tablet report, sending...
I20260928 06:42:32.181118 514 ts_manager.cc:194] Registered new tserver with Master: a34c9d02780440f68a330a63df8da0a9 (127.18.211.66:36929)
I20260928 06:42:32.181490 514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.66@KRBTEST.COM'} at 127.18.211.66:55893
WARNING: no policy specified for kudu/127.18.211.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab.
Entry for principal kudu/127.18.211.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab.
I20260928 06:42:32.197650 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.67:0
--local_ip_for_outbound_sockets=127.18.211.67
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:33269
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:42:32.276257 847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:42:32.276459 847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:42:32.276489 847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:42:32.277602 847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:42:32.277643 847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:42:32.277678 847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.67
I20260928 06:42:32.279102 847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:33269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.67
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:42:32.279335 847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:42:32.279534 847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:42:32.284230 855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:32.284240 853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:32.284250 852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:42:32.284543 847 server_base.cc:1034] running on GCE node
I20260928 06:42:32.284732 847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:42:32.284946 847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:42:32.286077 847 hybrid_clock.cc:648] HybridClock initialized: now 1790577752286069 us; error 26 us; skew 500 ppm
Sep 28 06:42:32 dist-test-slave-b28r krb5kdc[469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577752, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:42:32.288481 847 init.cc:377] Logged in from keytab as kudu/127.18.211.67@KRBTEST.COM (short username kudu)
I20260928 06:42:32.288916 847 webserver.cc:460] Webserver started at http://127.18.211.67:38399/ using document root <none> and password file <none>
I20260928 06:42:32.289112 847 fs_manager.cc:359] Metadata directory not provided
I20260928 06:42:32.289211 847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:42:32.289340 847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:42:32.290369 847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/data/instance:
uuid: "47100e01c7b84ed6b99fbebe03ed7e12"
format_stamp: "Formatted at 2026-09-28 06:42:32 on dist-test-slave-b28r"
server_key: "849b14553b2f722f4df6629ecebcf89e"
server_key_iv: "39d5a664f7d14d54167aceb03539f71b"
server_key_version: "encryptionkey@0"
I20260928 06:42:32.290788 847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance:
uuid: "47100e01c7b84ed6b99fbebe03ed7e12"
format_stamp: "Formatted at 2026-09-28 06:42:32 on dist-test-slave-b28r"
server_key: "849b14553b2f722f4df6629ecebcf89e"
server_key_iv: "39d5a664f7d14d54167aceb03539f71b"
server_key_version: "encryptionkey@0"
I20260928 06:42:32.292132 847 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:42:32.292822 863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:32.292958 847 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:32.293040 847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/wal
uuid: "47100e01c7b84ed6b99fbebe03ed7e12"
format_stamp: "Formatted at 2026-09-28 06:42:32 on dist-test-slave-b28r"
server_key: "849b14553b2f722f4df6629ecebcf89e"
server_key_iv: "39d5a664f7d14d54167aceb03539f71b"
server_key_version: "encryptionkey@0"
I20260928 06:42:32.293102 847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:42:32.329272 847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:42:32.331070 847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:42:32.331236 847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:42:32.331457 847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:42:32.331772 847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:42:32.331811 847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:32.331840 847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:42:32.331864 847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:32.337376 847 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.67:43049
I20260928 06:42:32.337430 976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.67:43049 every 8 connection(s)
I20260928 06:42:32.337716 847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
I20260928 06:42:32.342264 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 847
I20260928 06:42:32.342384 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance
I20260928 06:42:32.342584 19277 external_mini_cluster.cc:1428] Setting key aeb13e7f1105580567dc48b4e496d2b4
Sep 28 06:42:32 dist-test-slave-b28r krb5kdc[469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577752, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:42:32.346258 554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:32.338373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:38155 (local address 127.18.211.126:33269)
0928 06:42:32.338430 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:32.338432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:32.338648 (+ 216us) server_negotiation.cc:407] Connection header received
0928 06:42:32.339129 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:32.339131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:32.339141 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:32.339206 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:32.340075 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.340588 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.341511 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.341720 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.341771 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:32.343990 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:32.343991 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:32.343991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:32.343998 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:32.345072 (+ 1074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:32.345439 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:32.345439 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:32.345440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:32.345494 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:32.345701 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:32.345702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:32.345703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:32.345820 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:32.345881 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:32.346091 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 06:42:32.346165 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 06:42:32.346375 979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:32.338339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.67:38155)
0928 06:42:32.338503 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:32.338545 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:32.339028 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:32.339276 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:32.339280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:32.339814 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:42:32.339999 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:32.340003 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.340656 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.340657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:32.341441 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:32.341443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.341780 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.341781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:32.341872 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:32.342372 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:42:32.342382 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:42:32.343913 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:42:32.345136 (+ 1223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:32.345142 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:32.345144 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:32.345370 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:32.345543 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:32.345543 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:32.345543 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:32.345628 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:32.345875 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:42:32.345878 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:42:32.345983 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 06:42:32.346072 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 06:42:32.346183 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":79,"threads_started":1}
I20260928 06:42:32.346832 977 heartbeater.cc:347] Connected to a master server at 127.18.211.126:33269
I20260928 06:42:32.346907 977 heartbeater.cc:464] Registering TS with master...
I20260928 06:42:32.347062 977 heartbeater.cc:511] Master 127.18.211.126:33269 requested a full tablet report, sending...
I20260928 06:42:32.347427 514 ts_manager.cc:194] Registered new tserver with Master: 47100e01c7b84ed6b99fbebe03ed7e12 (127.18.211.67:43049)
I20260928 06:42:32.347767 514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.67@KRBTEST.COM'} at 127.18.211.67:38155
I20260928 06:42:32.354326 19277 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:42:32.361177 554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:32.355582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58022 (local address 127.18.211.126:33269)
0928 06:42:32.355620 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:32.355621 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:32.355751 (+ 130us) server_negotiation.cc:407] Connection header received
0928 06:42:32.355818 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:32.355819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:32.355829 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:32.355873 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:32.356396 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.356863 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.357583 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.357766 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.357815 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:32.358933 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:32.358935 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:32.358935 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:32.358942 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:32.360167 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:32.360467 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:32.360468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:32.360469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:32.360523 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:32.360682 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:32.360683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:32.360683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:32.360822 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:32.360908 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:32.361023 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 06:42:32.361092 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:42:32.362552 514 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:42:32.364230 514 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:42:32.370406 989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:32.366274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:36929 (local address 127.0.0.1:41804)
0928 06:42:32.366639 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:32.366647 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:32.366685 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:32.366976 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:32.366976 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:32.366978 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:42:32.367033 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:32.367035 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.368033 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.368034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:32.369255 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:32.369258 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.370108 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.370109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:32.370184 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:32.370200 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 06:42:32.370280 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 06:42:32.370332 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":63,"threads_started":1}
I20260928 06:42:32.370422 988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:32.366405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41804 (local address 127.18.211.66:36929)
0928 06:42:32.366822 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:32.366828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:32.366838 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:42:32.366876 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:32.366878 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:32.366892 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:32.367163 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:42:32.367242 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.367944 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.369326 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.370023 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.370097 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:32.370158 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:32.370296 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 06:42:32.370341 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":73,"threads_started":1}
I20260928 06:42:32.370728 990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:32.366635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35760 (local address 127.18.211.67:43049)
0928 06:42:32.366806 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:32.366810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:32.366819 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:42:32.366893 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:32.366895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:32.366909 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:32.366972 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:42:32.367184 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.368258 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.369478 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.370140 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.370198 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:32.370245 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:32.370603 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 06:42:32.370646 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":72,"threads_started":1}
I20260928 06:42:32.370863 986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:32.366471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:43049 (local address 127.0.0.1:35760)
0928 06:42:32.366778 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:32.366787 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:32.366822 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:32.367045 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:32.367046 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:32.367048 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:42:32.367110 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:32.367112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.368356 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.368356 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:32.369407 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:32.369409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.370281 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.370282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:32.370353 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:32.370358 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:42:32.370405 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 06:42:32.370446 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":43,"threads_started":1}
I20260928 06:42:32.371132 775 tablet_service.cc:1467] Processing CreateTablet for tablet ffc47ccb356b47fa99ac1e200ca0c1f8 (DEFAULT_TABLE table=test-table [id=eaf8683a74604d42b658d5e96d8a1140]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:42:32.371511 775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffc47ccb356b47fa99ac1e200ca0c1f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:42:32.371632 987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:32.366368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50382 (local address 127.18.211.65:42529)
0928 06:42:32.366549 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:32.366554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:32.366660 (+ 106us) server_negotiation.cc:407] Connection header received
0928 06:42:32.366753 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:32.366756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:32.366772 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:32.366857 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:42:32.367125 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.368253 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.370220 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.371240 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.371312 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:32.371369 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:32.371510 (+ 141us) server_negotiation.cc:299] Negotiation successful
0928 06:42:32.371558 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":82,"threads_started":1}
I20260928 06:42:32.371677 985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:32.366250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:42529 (local address 127.0.0.1:50382)
0928 06:42:32.366554 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:32.366596 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:32.366670 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:32.366946 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:32.366949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:32.366952 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:42:32.367018 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:32.367020 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.369476 (+ 2456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.369477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:32.370133 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:32.370135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.371376 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.371377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:32.371433 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:32.371437 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:42:32.371493 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 06:42:32.371534 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":80,"threads_started":1}
I20260928 06:42:32.372303 639 tablet_service.cc:1467] Processing CreateTablet for tablet ffc47ccb356b47fa99ac1e200ca0c1f8 (DEFAULT_TABLE table=test-table [id=eaf8683a74604d42b658d5e96d8a1140]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:42:32.372576 639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffc47ccb356b47fa99ac1e200ca0c1f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:42:32.374109 991 tablet_bootstrap.cc:492] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a34c9d02780440f68a330a63df8da0a9: Bootstrap starting.
I20260928 06:42:32.374624 911 tablet_service.cc:1467] Processing CreateTablet for tablet ffc47ccb356b47fa99ac1e200ca0c1f8 (DEFAULT_TABLE table=test-table [id=eaf8683a74604d42b658d5e96d8a1140]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:42:32.374769 991 tablet_bootstrap.cc:654] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a34c9d02780440f68a330a63df8da0a9: Neither blocks nor log segments found. Creating new log.
I20260928 06:42:32.374899 911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffc47ccb356b47fa99ac1e200ca0c1f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:42:32.375252 993 tablet_bootstrap.cc:492] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a8a75496766348418bca6d243610c5dc: Bootstrap starting.
I20260928 06:42:32.375366 991 log.cc:824] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a34c9d02780440f68a330a63df8da0a9: Log is configured to *not* fsync() on all Append() calls
I20260928 06:42:32.375973 993 tablet_bootstrap.cc:654] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a8a75496766348418bca6d243610c5dc: Neither blocks nor log segments found. Creating new log.
I20260928 06:42:32.376111 991 tablet_bootstrap.cc:492] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a34c9d02780440f68a330a63df8da0a9: No bootstrap required, opened a new log
I20260928 06:42:32.376178 991 ts_tablet_manager.cc:1397] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a34c9d02780440f68a330a63df8da0a9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:42:32.376529 993 log.cc:824] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a8a75496766348418bca6d243610c5dc: Log is configured to *not* fsync() on all Append() calls
I20260928 06:42:32.377295 993 tablet_bootstrap.cc:492] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a8a75496766348418bca6d243610c5dc: No bootstrap required, opened a new log
I20260928 06:42:32.377363 993 ts_tablet_manager.cc:1397] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a8a75496766348418bca6d243610c5dc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:42:32.377743 991 raft_consensus.cc:348] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a34c9d02780440f68a330a63df8da0a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a75496766348418bca6d243610c5dc" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 42529 } } peers { permanent_uuid: "47100e01c7b84ed6b99fbebe03ed7e12" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 43049 } } peers { permanent_uuid: "a34c9d02780440f68a330a63df8da0a9" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36929 } }
I20260928 06:42:32.377902 991 raft_consensus.cc:374] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a34c9d02780440f68a330a63df8da0a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:42:32.377942 991 raft_consensus.cc:729] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a34c9d02780440f68a330a63df8da0a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a34c9d02780440f68a330a63df8da0a9, State: Initialized, Role: FOLLOWER
I20260928 06:42:32.378068 996 tablet_bootstrap.cc:492] T ffc47ccb356b47fa99ac1e200ca0c1f8 P 47100e01c7b84ed6b99fbebe03ed7e12: Bootstrap starting.
I20260928 06:42:32.378075 991 consensus_queue.cc:260] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a34c9d02780440f68a330a63df8da0a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a75496766348418bca6d243610c5dc" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 42529 } } peers { permanent_uuid: "47100e01c7b84ed6b99fbebe03ed7e12" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 43049 } } peers { permanent_uuid: "a34c9d02780440f68a330a63df8da0a9" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36929 } }
I20260928 06:42:32.378381 991 ts_tablet_manager.cc:1428] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a34c9d02780440f68a330a63df8da0a9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:42:32.378513 842 heartbeater.cc:503] Master 127.18.211.126:33269 was elected leader, sending a full tablet report...
I20260928 06:42:32.378667 996 tablet_bootstrap.cc:654] T ffc47ccb356b47fa99ac1e200ca0c1f8 P 47100e01c7b84ed6b99fbebe03ed7e12: Neither blocks nor log segments found. Creating new log.
I20260928 06:42:32.379318 996 log.cc:824] T ffc47ccb356b47fa99ac1e200ca0c1f8 P 47100e01c7b84ed6b99fbebe03ed7e12: Log is configured to *not* fsync() on all Append() calls
I20260928 06:42:32.379225 993 raft_consensus.cc:348] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a8a75496766348418bca6d243610c5dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a75496766348418bca6d243610c5dc" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 42529 } } peers { permanent_uuid: "47100e01c7b84ed6b99fbebe03ed7e12" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 43049 } } peers { permanent_uuid: "a34c9d02780440f68a330a63df8da0a9" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36929 } }
I20260928 06:42:32.379379 993 raft_consensus.cc:374] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a8a75496766348418bca6d243610c5dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:42:32.379415 993 raft_consensus.cc:729] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a8a75496766348418bca6d243610c5dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8a75496766348418bca6d243610c5dc, State: Initialized, Role: FOLLOWER
I20260928 06:42:32.379526 993 consensus_queue.cc:260] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a8a75496766348418bca6d243610c5dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a75496766348418bca6d243610c5dc" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 42529 } } peers { permanent_uuid: "47100e01c7b84ed6b99fbebe03ed7e12" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 43049 } } peers { permanent_uuid: "a34c9d02780440f68a330a63df8da0a9" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36929 } }
I20260928 06:42:32.379792 993 ts_tablet_manager.cc:1428] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a8a75496766348418bca6d243610c5dc: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:42:32.379920 706 heartbeater.cc:503] Master 127.18.211.126:33269 was elected leader, sending a full tablet report...
I20260928 06:42:32.379974 996 tablet_bootstrap.cc:492] T ffc47ccb356b47fa99ac1e200ca0c1f8 P 47100e01c7b84ed6b99fbebe03ed7e12: No bootstrap required, opened a new log
I20260928 06:42:32.380034 996 ts_tablet_manager.cc:1397] T ffc47ccb356b47fa99ac1e200ca0c1f8 P 47100e01c7b84ed6b99fbebe03ed7e12: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 06:42:32.381418 996 raft_consensus.cc:348] T ffc47ccb356b47fa99ac1e200ca0c1f8 P 47100e01c7b84ed6b99fbebe03ed7e12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a75496766348418bca6d243610c5dc" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 42529 } } peers { permanent_uuid: "47100e01c7b84ed6b99fbebe03ed7e12" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 43049 } } peers { permanent_uuid: "a34c9d02780440f68a330a63df8da0a9" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36929 } }
I20260928 06:42:32.381536 996 raft_consensus.cc:374] T ffc47ccb356b47fa99ac1e200ca0c1f8 P 47100e01c7b84ed6b99fbebe03ed7e12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:42:32.381556 996 raft_consensus.cc:729] T ffc47ccb356b47fa99ac1e200ca0c1f8 P 47100e01c7b84ed6b99fbebe03ed7e12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47100e01c7b84ed6b99fbebe03ed7e12, State: Initialized, Role: FOLLOWER
I20260928 06:42:32.381654 996 consensus_queue.cc:260] T ffc47ccb356b47fa99ac1e200ca0c1f8 P 47100e01c7b84ed6b99fbebe03ed7e12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a75496766348418bca6d243610c5dc" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 42529 } } peers { permanent_uuid: "47100e01c7b84ed6b99fbebe03ed7e12" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 43049 } } peers { permanent_uuid: "a34c9d02780440f68a330a63df8da0a9" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36929 } }
I20260928 06:42:32.381881 996 ts_tablet_manager.cc:1428] T ffc47ccb356b47fa99ac1e200ca0c1f8 P 47100e01c7b84ed6b99fbebe03ed7e12: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:42:32.381909 977 heartbeater.cc:503] Master 127.18.211.126:33269 was elected leader, sending a full tablet report...
I20260928 06:42:32.385285 997 raft_consensus.cc:482] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a34c9d02780440f68a330a63df8da0a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:42:32.385360 997 raft_consensus.cc:504] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a34c9d02780440f68a330a63df8da0a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a75496766348418bca6d243610c5dc" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 42529 } } peers { permanent_uuid: "47100e01c7b84ed6b99fbebe03ed7e12" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 43049 } } peers { permanent_uuid: "a34c9d02780440f68a330a63df8da0a9" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36929 } }
I20260928 06:42:32.385653 997 leader_election.cc:288] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a34c9d02780440f68a330a63df8da0a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a8a75496766348418bca6d243610c5dc (127.18.211.65:42529), 47100e01c7b84ed6b99fbebe03ed7e12 (127.18.211.67:43049)
I20260928 06:42:32.388252 987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:32.385888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:55347 (local address 127.18.211.65:42529)
0928 06:42:32.385990 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:32.385992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:32.385999 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:42:32.386019 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:32.386020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:32.386029 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:32.386076 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:42:32.386196 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.386651 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.387572 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.387955 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.388015 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:32.388023 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:32.388168 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 06:42:32.388199 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 06:42:32.388250 844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:32.385803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:42529 (local address 127.18.211.66:55347)
0928 06:42:32.385853 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:32.385865 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:32.385904 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:32.386080 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:32.386081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:32.386082 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:42:32.386142 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:32.386143 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.386718 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.386719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:32.387500 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:32.387502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.388021 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.388022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:32.388096 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:32.388099 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:42:32.388143 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 06:42:32.388181 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":5}
I20260928 06:42:32.388377 990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:32.385984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:37599 (local address 127.18.211.67:43049)
0928 06:42:32.386026 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:32.386028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:32.386110 (+ 82us) server_negotiation.cc:407] Connection header received
0928 06:42:32.386155 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:32.386156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:32.386164 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:32.386213 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:42:32.386364 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.386821 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.387720 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.388088 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.388148 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:32.388155 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:32.388288 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 06:42:32.388321 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:42:32.388388 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:32.385928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:43049 (local address 127.18.211.66:37599)
0928 06:42:32.386047 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:32.386056 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:32.386111 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:32.386250 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:32.386250 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:32.386252 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:42:32.386313 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:32.386314 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.386885 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.386886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:32.387644 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:32.387646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.388155 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.388156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:32.388232 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:32.388235 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:42:32.388278 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 06:42:32.388313 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
I20260928 06:42:32.388722 931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc47ccb356b47fa99ac1e200ca0c1f8" candidate_uuid: "a34c9d02780440f68a330a63df8da0a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47100e01c7b84ed6b99fbebe03ed7e12" is_pre_election: true
I20260928 06:42:32.388767 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc47ccb356b47fa99ac1e200ca0c1f8" candidate_uuid: "a34c9d02780440f68a330a63df8da0a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8a75496766348418bca6d243610c5dc" is_pre_election: true
I20260928 06:42:32.388861 931 raft_consensus.cc:2454] T ffc47ccb356b47fa99ac1e200ca0c1f8 P 47100e01c7b84ed6b99fbebe03ed7e12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a34c9d02780440f68a330a63df8da0a9 in term 0.
I20260928 06:42:32.388900 659 raft_consensus.cc:2454] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a8a75496766348418bca6d243610c5dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a34c9d02780440f68a330a63df8da0a9 in term 0.
I20260928 06:42:32.389050 730 leader_election.cc:302] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a34c9d02780440f68a330a63df8da0a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47100e01c7b84ed6b99fbebe03ed7e12, a34c9d02780440f68a330a63df8da0a9; no voters:
I20260928 06:42:32.389156 997 raft_consensus.cc:2781] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a34c9d02780440f68a330a63df8da0a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:42:32.389215 997 raft_consensus.cc:482] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a34c9d02780440f68a330a63df8da0a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:42:32.389242 997 raft_consensus.cc:3037] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a34c9d02780440f68a330a63df8da0a9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:42:32.389837 997 raft_consensus.cc:504] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a34c9d02780440f68a330a63df8da0a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a75496766348418bca6d243610c5dc" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 42529 } } peers { permanent_uuid: "47100e01c7b84ed6b99fbebe03ed7e12" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 43049 } } peers { permanent_uuid: "a34c9d02780440f68a330a63df8da0a9" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36929 } }
I20260928 06:42:32.389935 997 leader_election.cc:288] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a34c9d02780440f68a330a63df8da0a9 [CANDIDATE]: Term 1 election: Requested vote from peers a8a75496766348418bca6d243610c5dc (127.18.211.65:42529), 47100e01c7b84ed6b99fbebe03ed7e12 (127.18.211.67:43049)
I20260928 06:42:32.390094 931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc47ccb356b47fa99ac1e200ca0c1f8" candidate_uuid: "a34c9d02780440f68a330a63df8da0a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47100e01c7b84ed6b99fbebe03ed7e12"
I20260928 06:42:32.390094 659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc47ccb356b47fa99ac1e200ca0c1f8" candidate_uuid: "a34c9d02780440f68a330a63df8da0a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8a75496766348418bca6d243610c5dc"
I20260928 06:42:32.390159 931 raft_consensus.cc:3037] T ffc47ccb356b47fa99ac1e200ca0c1f8 P 47100e01c7b84ed6b99fbebe03ed7e12 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:42:32.390161 659 raft_consensus.cc:3037] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a8a75496766348418bca6d243610c5dc [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:42:32.390720 931 raft_consensus.cc:2454] T ffc47ccb356b47fa99ac1e200ca0c1f8 P 47100e01c7b84ed6b99fbebe03ed7e12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a34c9d02780440f68a330a63df8da0a9 in term 1.
I20260928 06:42:32.390865 659 raft_consensus.cc:2454] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a8a75496766348418bca6d243610c5dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a34c9d02780440f68a330a63df8da0a9 in term 1.
I20260928 06:42:32.390869 730 leader_election.cc:302] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a34c9d02780440f68a330a63df8da0a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47100e01c7b84ed6b99fbebe03ed7e12, a34c9d02780440f68a330a63df8da0a9; no voters:
I20260928 06:42:32.390945 997 raft_consensus.cc:2781] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a34c9d02780440f68a330a63df8da0a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:42:32.391124 997 raft_consensus.cc:686] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a34c9d02780440f68a330a63df8da0a9 [term 1 LEADER]: Becoming Leader. State: Replica: a34c9d02780440f68a330a63df8da0a9, State: Running, Role: LEADER
I20260928 06:42:32.391219 997 consensus_queue.cc:237] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a34c9d02780440f68a330a63df8da0a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a75496766348418bca6d243610c5dc" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 42529 } } peers { permanent_uuid: "47100e01c7b84ed6b99fbebe03ed7e12" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 43049 } } peers { permanent_uuid: "a34c9d02780440f68a330a63df8da0a9" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36929 } }
I20260928 06:42:32.391916 514 catalog_manager.cc:5495] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a34c9d02780440f68a330a63df8da0a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a34c9d02780440f68a330a63df8da0a9 (127.18.211.66). New cstate: current_term: 1 leader_uuid: "a34c9d02780440f68a330a63df8da0a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a75496766348418bca6d243610c5dc" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 42529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47100e01c7b84ed6b99fbebe03ed7e12" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 43049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a34c9d02780440f68a330a63df8da0a9" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36929 } health_report { overall_health: HEALTHY } } }
I20260928 06:42:32.398398 988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:32.396433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41812 (local address 127.18.211.66:36929)
0928 06:42:32.396468 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:32.396470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:32.396477 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:42:32.396521 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:32.396522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:32.396537 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:32.396582 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:42:32.396673 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.397135 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.397816 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.397995 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.398043 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:32.398127 (+ 84us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:42:32.398213 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:42:32.398305 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:32.398318 (+ 13us) server_negotiation.cc:299] Negotiation successful
0928 06:42:32.398333 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:42:32.400056 931 raft_consensus.cc:1261] T ffc47ccb356b47fa99ac1e200ca0c1f8 P 47100e01c7b84ed6b99fbebe03ed7e12 [term 1 FOLLOWER]: Refusing update from remote peer a34c9d02780440f68a330a63df8da0a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:42:32.400056 659 raft_consensus.cc:1261] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a8a75496766348418bca6d243610c5dc [term 1 FOLLOWER]: Refusing update from remote peer a34c9d02780440f68a330a63df8da0a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:42:32.400396 1002 consensus_queue.cc:1035] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a34c9d02780440f68a330a63df8da0a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a75496766348418bca6d243610c5dc" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 42529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:42:32.400506 997 consensus_queue.cc:1035] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a34c9d02780440f68a330a63df8da0a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47100e01c7b84ed6b99fbebe03ed7e12" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 43049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:42:32.403270 1007 mvcc.cc:205] Tried to move back new op lower bound from 7334206473828352000 to 7334206473795006464. Current Snapshot: MvccSnapshot[applied={T|T < 7334206473828352000 or (T in {7334206473828352000})}]
I20260928 06:42:32.405104 513 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:42:32.405184 513 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:42:32.405848 513 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e: Sending DeleteTablet for 3 replicas of tablet ffc47ccb356b47fa99ac1e200ca0c1f8
I20260928 06:42:32.406154 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 576
I20260928 06:42:32.406136 775 tablet_service.cc:1514] Processing DeleteTablet for tablet ffc47ccb356b47fa99ac1e200ca0c1f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:42:32 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:41804
I20260928 06:42:32.406263 639 tablet_service.cc:1514] Processing DeleteTablet for tablet ffc47ccb356b47fa99ac1e200ca0c1f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:42:32 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:50382
I20260928 06:42:32.406637 911 tablet_service.cc:1514] Processing DeleteTablet for tablet ffc47ccb356b47fa99ac1e200ca0c1f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:42:32 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:35760
I20260928 06:42:32.406699 1014 tablet_replica.cc:330] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a34c9d02780440f68a330a63df8da0a9: stopping tablet replica
I20260928 06:42:32.406802 1014 raft_consensus.cc:2229] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a34c9d02780440f68a330a63df8da0a9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:42:32.406896 1014 raft_consensus.cc:2258] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a34c9d02780440f68a330a63df8da0a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:42:32.406898 1015 tablet_replica.cc:330] T ffc47ccb356b47fa99ac1e200ca0c1f8 P 47100e01c7b84ed6b99fbebe03ed7e12: stopping tablet replica
I20260928 06:42:32.407058 1015 raft_consensus.cc:2229] T ffc47ccb356b47fa99ac1e200ca0c1f8 P 47100e01c7b84ed6b99fbebe03ed7e12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:42:32.407212 1015 raft_consensus.cc:2258] T ffc47ccb356b47fa99ac1e200ca0c1f8 P 47100e01c7b84ed6b99fbebe03ed7e12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:42:32.407462 1014 ts_tablet_manager.cc:1905] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a34c9d02780440f68a330a63df8da0a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:42:32.407682 1015 ts_tablet_manager.cc:1905] T ffc47ccb356b47fa99ac1e200ca0c1f8 P 47100e01c7b84ed6b99fbebe03ed7e12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:42:32.409056 1014 ts_tablet_manager.cc:1918] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a34c9d02780440f68a330a63df8da0a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:42:32.409135 1014 log.cc:1196] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a34c9d02780440f68a330a63df8da0a9: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/wal/wals/ffc47ccb356b47fa99ac1e200ca0c1f8
I20260928 06:42:32.409305 1015 ts_tablet_manager.cc:1918] T ffc47ccb356b47fa99ac1e200ca0c1f8 P 47100e01c7b84ed6b99fbebe03ed7e12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:42:32.409418 1015 log.cc:1196] T ffc47ccb356b47fa99ac1e200ca0c1f8 P 47100e01c7b84ed6b99fbebe03ed7e12: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/wal/wals/ffc47ccb356b47fa99ac1e200ca0c1f8
I20260928 06:42:32.409468 1014 ts_tablet_manager.cc:1939] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a34c9d02780440f68a330a63df8da0a9: Deleting consensus metadata
I20260928 06:42:32.409770 1015 ts_tablet_manager.cc:1939] T ffc47ccb356b47fa99ac1e200ca0c1f8 P 47100e01c7b84ed6b99fbebe03ed7e12: Deleting consensus metadata
I20260928 06:42:32.409852 498 catalog_manager.cc:4841] TS a34c9d02780440f68a330a63df8da0a9 (127.18.211.66:36929): tablet ffc47ccb356b47fa99ac1e200ca0c1f8 (table test-table [id=eaf8683a74604d42b658d5e96d8a1140]) successfully deleted
I20260928 06:42:32.410248 500 catalog_manager.cc:4841] TS 47100e01c7b84ed6b99fbebe03ed7e12 (127.18.211.67:43049): tablet ffc47ccb356b47fa99ac1e200ca0c1f8 (table test-table [id=eaf8683a74604d42b658d5e96d8a1140]) successfully deleted
W20260928 06:42:32.412747 500 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.211.65:42529 (error 108)
I20260928 06:42:32.413220 985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:32.413093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:42529 (local address 127.0.0.1:50396)
0928 06:42:32.413149 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:32.413188 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.65:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 06:42:32.413233 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 712
W20260928 06:42:32.414444 500 catalog_manager.cc:4584] TS a8a75496766348418bca6d243610c5dc (127.18.211.65:42529): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ffc47ccb356b47fa99ac1e200ca0c1f8: Network error: Client connection negotiation failed: client connection to 127.18.211.65:42529: connect: Connection refused (error 111)
I20260928 06:42:32.418197 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 847
I20260928 06:42:32.422297 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 482
I20260928 06:42:32.426858 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:33269
--webserver_interface=127.18.211.126
--webserver_port=34409
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:33269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:42:32.506292 1017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:42:32.506494 1017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:42:32.506515 1017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:42:32.507642 1017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:42:32.507673 1017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:42:32.507686 1017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:42:32.507696 1017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:42:32.507707 1017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:42:32.509122 1017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:33269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:33269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=34409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:42:32.509308 1017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:42:32.509475 1017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:42:32.517303 1024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:32.517309 1023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:32.517310 1026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:42:32.517611 1017 server_base.cc:1034] running on GCE node
I20260928 06:42:32.517874 1017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:42:32.518177 1017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:42:32.519305 1017 hybrid_clock.cc:648] HybridClock initialized: now 1790577752519288 us; error 32 us; skew 500 ppm
Sep 28 06:42:32 dist-test-slave-b28r krb5kdc[469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577752, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:42:32.521754 1017 init.cc:377] Logged in from keytab as kudu/127.18.211.126@KRBTEST.COM (short username kudu)
I20260928 06:42:32.522205 1017 webserver.cc:460] Webserver started at http://127.18.211.126:34409/ using document root <none> and password file <none>
I20260928 06:42:32.522418 1017 fs_manager.cc:359] Metadata directory not provided
I20260928 06:42:32.522480 1017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:42:32.523854 1017 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:42:32.524472 1033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:32.524633 1017 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:42:32.524721 1017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "98ea104104c0443bbfad48f3cd5f272e"
format_stamp: "Formatted at 2026-09-28 06:42:31 on dist-test-slave-b28r"
server_key: "56564d69f13ab234aa9b81c472c2b297"
server_key_iv: "9f14f35f2f6196e399d4e389b3273995"
server_key_version: "encryptionkey@0"
I20260928 06:42:32.525110 1017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:42:32.554124 1017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:42:32.556103 1017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:42:32.556244 1017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:42:32.560001 1017 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:33269
I20260928 06:42:32.560047 1085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:33269 every 8 connection(s)
I20260928 06:42:32.560411 1017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:42:32.561868 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 1017
I20260928 06:42:32.562001 1086 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:42:32.562131 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.65:42529
--local_ip_for_outbound_sockets=127.18.211.65
--tserver_master_addrs=127.18.211.126:33269
--webserver_port=46339
--webserver_interface=127.18.211.65
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:42:32.562734 1086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e: Bootstrap starting.
I20260928 06:42:32.571223 1086 log.cc:824] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e: Log is configured to *not* fsync() on all Append() calls
I20260928 06:42:32.572996 1086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:42:32.573271 1086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e: Bootstrap complete.
I20260928 06:42:32.575475 1086 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ea104104c0443bbfad48f3cd5f272e" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 33269 } }
I20260928 06:42:32.575706 1086 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98ea104104c0443bbfad48f3cd5f272e, State: Initialized, Role: FOLLOWER
I20260928 06:42:32.575853 1086 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ea104104c0443bbfad48f3cd5f272e" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 33269 } }
I20260928 06:42:32.575909 1086 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:42:32.575953 1086 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:42:32.575999 1086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:42:32.576975 1086 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ea104104c0443bbfad48f3cd5f272e" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 33269 } }
I20260928 06:42:32.577064 1086 leader_election.cc:302] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98ea104104c0443bbfad48f3cd5f272e; no voters:
I20260928 06:42:32.577244 1086 leader_election.cc:288] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 06:42:32.577291 1090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:42:32.577409 1090 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [term 2 LEADER]: Becoming Leader. State: Replica: 98ea104104c0443bbfad48f3cd5f272e, State: Running, Role: LEADER
I20260928 06:42:32.577499 1086 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:42:32.577492 1090 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ea104104c0443bbfad48f3cd5f272e" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 33269 } }
I20260928 06:42:32.577805 1090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98ea104104c0443bbfad48f3cd5f272e. Latest consensus state: current_term: 2 leader_uuid: "98ea104104c0443bbfad48f3cd5f272e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ea104104c0443bbfad48f3cd5f272e" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 33269 } } }
I20260928 06:42:32.577888 1090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [sys.catalog]: This master's current role is: LEADER
I20260928 06:42:32.578133 1092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "98ea104104c0443bbfad48f3cd5f272e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ea104104c0443bbfad48f3cd5f272e" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 33269 } } }
I20260928 06:42:32.578236 1092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [sys.catalog]: This master's current role is: LEADER
I20260928 06:42:32.578364 1096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:42:32.579191 1096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:42:32.579298 1096 catalog_manager.cc:1259] Loaded cluster ID: 548c21bb1d884ab1bacc55b42743b6d3
I20260928 06:42:32.579332 1096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:42:32.579632 1096 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 06:42:32.584276 1096 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 06:42:32.586669 1096 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:42:32.586879 1096 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e: Loaded TSK: 0
I20260928 06:42:32.587149 1096 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:42:32.643401 1088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:42:32.643599 1088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:42:32.643631 1088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:42:32.644733 1088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:42:32.644776 1088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:42:32.644811 1088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.65
I20260928 06:42:32.646221 1088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.65:42529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.211.65
--webserver_port=46339
--tserver_master_addrs=127.18.211.126:33269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.65
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:42:32.646445 1088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:42:32.646647 1088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:42:32.651418 1116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:32.651363 1113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:32.651449 1114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:42:32.651793 1088 server_base.cc:1034] running on GCE node
I20260928 06:42:32.651996 1088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:42:32.652218 1088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:42:32.653357 1088 hybrid_clock.cc:648] HybridClock initialized: now 1790577752653345 us; error 30 us; skew 500 ppm
Sep 28 06:42:32 dist-test-slave-b28r krb5kdc[469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577752, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:42:32.655964 1088 init.cc:377] Logged in from keytab as kudu/127.18.211.65@KRBTEST.COM (short username kudu)
I20260928 06:42:32.656404 1088 webserver.cc:460] Webserver started at http://127.18.211.65:46339/ using document root <none> and password file <none>
I20260928 06:42:32.656615 1088 fs_manager.cc:359] Metadata directory not provided
I20260928 06:42:32.656675 1088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:42:32.658010 1088 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:42:32.658679 1123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:32.658854 1088 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:42:32.658955 1088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/wal
uuid: "a8a75496766348418bca6d243610c5dc"
format_stamp: "Formatted at 2026-09-28 06:42:31 on dist-test-slave-b28r"
server_key: "2938121db54c2b6505cee2d6a936f3d6"
server_key_iv: "0909d0eaf6362a26fda324f17f0aa3ce"
server_key_version: "encryptionkey@0"
I20260928 06:42:32.659411 1088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:42:32.681536 1088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:42:32.683944 1088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:42:32.684100 1088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:42:32.684356 1088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:42:32.684854 1130 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 06:42:32.686178 1088 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 06:42:32.686239 1088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 06:42:32.686272 1088 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 06:42:32.687168 1088 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 06:42:32.687214 1088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260928 06:42:32.687270 1130 tablet_bootstrap.cc:492] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a8a75496766348418bca6d243610c5dc: Bootstrap starting.
I20260928 06:42:32.694152 1088 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.65:42529
I20260928 06:42:32.694379 1238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.65:42529 every 8 connection(s)
I20260928 06:42:32.694622 1088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
I20260928 06:42:32.696913 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 1088
I20260928 06:42:32.697249 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.66:36929
--local_ip_for_outbound_sockets=127.18.211.66
--tserver_master_addrs=127.18.211.126:33269
--webserver_port=38643
--webserver_interface=127.18.211.66
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:42:32.697549 1130 log.cc:824] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a8a75496766348418bca6d243610c5dc: Log is configured to *not* fsync() on all Append() calls
I20260928 06:42:32.700362 1130 tablet_bootstrap.cc:492] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a8a75496766348418bca6d243610c5dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:42:32.700629 1130 tablet_bootstrap.cc:492] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a8a75496766348418bca6d243610c5dc: Bootstrap complete.
I20260928 06:42:32.701718 1130 ts_tablet_manager.cc:1397] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a8a75496766348418bca6d243610c5dc: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.007s
I20260928 06:42:32.703107 1130 raft_consensus.cc:348] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a8a75496766348418bca6d243610c5dc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a75496766348418bca6d243610c5dc" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 42529 } } peers { permanent_uuid: "47100e01c7b84ed6b99fbebe03ed7e12" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 43049 } } peers { permanent_uuid: "a34c9d02780440f68a330a63df8da0a9" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36929 } }
I20260928 06:42:32.703428 1130 raft_consensus.cc:729] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a8a75496766348418bca6d243610c5dc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8a75496766348418bca6d243610c5dc, State: Initialized, Role: FOLLOWER
I20260928 06:42:32.703711 1130 consensus_queue.cc:260] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a8a75496766348418bca6d243610c5dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a8a75496766348418bca6d243610c5dc" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 42529 } } peers { permanent_uuid: "47100e01c7b84ed6b99fbebe03ed7e12" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 43049 } } peers { permanent_uuid: "a34c9d02780440f68a330a63df8da0a9" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36929 } }
I20260928 06:42:32.704071 1130 ts_tablet_manager.cc:1428] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a8a75496766348418bca6d243610c5dc: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
Sep 28 06:42:32 dist-test-slave-b28r krb5kdc[469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577752, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:42:32.710734 1241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:32.695375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.65:44611)
0928 06:42:32.697818 (+ 2443us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:32.697861 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:32.699398 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:32.699930 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:32.699935 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:32.700268 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:42:32.700524 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:32.700530 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.702161 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.702162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:32.703429 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:32.703433 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.704298 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.704298 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:32.704433 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:32.705025 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:42:32.705034 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:42:32.706792 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:42:32.708948 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:32.708956 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:32.708958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:32.709265 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:32.709530 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:32.709531 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:32.709531 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:32.709667 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:32.710083 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:42:32.710089 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:42:32.710238 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 06:42:32.710353 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 06:42:32.710510 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2399,"thread_start_us":103,"threads_started":1}
I20260928 06:42:32.710772 1242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:32.695481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:44611 (local address 127.18.211.126:33269)
0928 06:42:32.698846 (+ 3365us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:32.698855 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:32.698899 (+ 44us) server_negotiation.cc:407] Connection header received
0928 06:42:32.699672 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:32.699679 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:32.699742 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:32.699874 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:32.700659 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.702052 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.703520 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.704117 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.704207 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:32.706907 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:32.706912 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:32.706913 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:32.706926 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:32.708855 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:32.709347 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:32.709350 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:32.709351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:32.709461 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:32.709734 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:32.709735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:32.709735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:32.710011 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:32.710086 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:32.710407 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 06:42:32.710575 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3309,"thread_start_us":98,"threads_started":1}
I20260928 06:42:32.712288 1239 heartbeater.cc:347] Connected to a master server at 127.18.211.126:33269
I20260928 06:42:32.712380 1239 heartbeater.cc:464] Registering TS with master...
I20260928 06:42:32.712589 1239 heartbeater.cc:511] Master 127.18.211.126:33269 requested a full tablet report, sending...
I20260928 06:42:32.713281 1050 ts_manager.cc:194] Registered new tserver with Master: a8a75496766348418bca6d243610c5dc (127.18.211.65:42529)
I20260928 06:42:32.714396 1050 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.65@KRBTEST.COM'} at 127.18.211.65:44611
I20260928 06:42:32.715912 1239 heartbeater.cc:503] Master 127.18.211.126:33269 was elected leader, sending a full tablet report...
Sep 28 06:42:32 dist-test-slave-b28r krb5kdc[469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577752, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for kudu/127.18.211.65@KRBTEST.COM
I20260928 06:42:32.723743 1246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:32.713858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:42529 (local address 127.0.0.1:50406)
0928 06:42:32.714227 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:32.714260 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:32.714320 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:32.714589 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:32.714594 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:32.716124 (+ 1530us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:42:32.716183 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:32.716186 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.717452 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.717453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:32.718389 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:32.718392 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.718880 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.718881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:32.718962 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:32.719400 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:42:32.719405 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:42:32.720931 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:42:32.722636 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:32.722644 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:32.722646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:32.722860 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:32.723051 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:32.723052 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:32.723052 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:32.723137 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:32.723395 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:42:32.723402 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:42:32.723455 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 06:42:32.723519 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 06:42:32.723632 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":87,"threads_started":1}
I20260928 06:42:32.723747 1247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:32.714087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50406 (local address 127.18.211.65:42529)
0928 06:42:32.714429 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:32.714433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:32.714442 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:42:32.714471 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:32.714473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:32.714485 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:32.714537 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:32.716272 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.717363 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.718491 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.718792 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.718866 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:32.721014 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:32.721023 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:32.721024 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:32.721031 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:32.722569 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:32.722934 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:32.722937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:32.722937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:32.722993 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:32.723192 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:32.723193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:32.723193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:32.723333 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:32.723403 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:32.723577 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 06:42:32.723665 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":64,"threads_started":1}
I20260928 06:42:32.724094 1173 tablet_service.cc:1514] Processing DeleteTablet for tablet ffc47ccb356b47fa99ac1e200ca0c1f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:42:32 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:50406
I20260928 06:42:32.724589 1248 tablet_replica.cc:330] stopping tablet replica
I20260928 06:42:32.724697 1248 raft_consensus.cc:2229] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a8a75496766348418bca6d243610c5dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:42:32.724772 1248 raft_consensus.cc:2258] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a8a75496766348418bca6d243610c5dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:42:32.725066 1248 ts_tablet_manager.cc:1905] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a8a75496766348418bca6d243610c5dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:42:32.727170 1248 ts_tablet_manager.cc:1918] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a8a75496766348418bca6d243610c5dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:42:32.727244 1248 log.cc:1196] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a8a75496766348418bca6d243610c5dc: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/wal/wals/ffc47ccb356b47fa99ac1e200ca0c1f8
I20260928 06:42:32.727455 1248 ts_tablet_manager.cc:1939] T ffc47ccb356b47fa99ac1e200ca0c1f8 P a8a75496766348418bca6d243610c5dc: Deleting consensus metadata
I20260928 06:42:32.727942 1036 catalog_manager.cc:4841] TS a8a75496766348418bca6d243610c5dc (127.18.211.65:42529): tablet ffc47ccb356b47fa99ac1e200ca0c1f8 (table test-table [id=eaf8683a74604d42b658d5e96d8a1140]) successfully deleted
W20260928 06:42:32.784595 1244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:42:32.784788 1244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:42:32.784809 1244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:42:32.785928 1244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:42:32.785969 1244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:42:32.785992 1244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.66
I20260928 06:42:32.787415 1244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.66:36929
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.211.66
--webserver_port=38643
--tserver_master_addrs=127.18.211.126:33269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.66
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:42:32.787617 1244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:42:32.787849 1244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:42:32.792699 1244 server_base.cc:1034] running on GCE node
W20260928 06:42:32.792675 1253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:32.792678 1256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:32.792670 1254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:42:32.793120 1244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:42:32.793402 1244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:42:32.794553 1244 hybrid_clock.cc:648] HybridClock initialized: now 1790577752794537 us; error 27 us; skew 500 ppm
Sep 28 06:42:32 dist-test-slave-b28r krb5kdc[469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577752, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:42:32.796968 1244 init.cc:377] Logged in from keytab as kudu/127.18.211.66@KRBTEST.COM (short username kudu)
I20260928 06:42:32.797410 1244 webserver.cc:460] Webserver started at http://127.18.211.66:38643/ using document root <none> and password file <none>
I20260928 06:42:32.797601 1244 fs_manager.cc:359] Metadata directory not provided
I20260928 06:42:32.797664 1244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:42:32.798931 1244 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:42:32.799547 1263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:32.799690 1244 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:32.799769 1244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/wal
uuid: "a34c9d02780440f68a330a63df8da0a9"
format_stamp: "Formatted at 2026-09-28 06:42:32 on dist-test-slave-b28r"
server_key: "bbf315a9465a08108bd594d407748d18"
server_key_iv: "929f02d6a143257417e06b4f4b5e5f14"
server_key_version: "encryptionkey@0"
I20260928 06:42:32.800093 1244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:42:32.815713 1244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:42:32.817519 1244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:42:32.817653 1244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:42:32.817874 1244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:42:32.818169 1244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:42:32.818202 1244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:32.818229 1244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:42:32.818282 1244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:32.824117 1244 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.66:36929
I20260928 06:42:32.824164 1377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.66:36929 every 8 connection(s)
I20260928 06:42:32.824499 1244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:42:32 dist-test-slave-b28r krb5kdc[469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577752, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:42:32.833105 1242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:32.825189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:47179 (local address 127.18.211.126:33269)
0928 06:42:32.825346 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:32.825348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:32.825432 (+ 84us) server_negotiation.cc:407] Connection header received
0928 06:42:32.825924 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:32.825925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:32.825938 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:32.825989 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:32.826985 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.827537 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.828570 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.828777 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.828842 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:32.830840 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:32.830842 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:32.830842 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:32.830848 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:32.832024 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:32.832418 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:32.832418 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:32.832419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:32.832467 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:32.832643 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:32.832644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:32.832644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:32.832732 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:32.832774 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:32.832974 (+ 200us) server_negotiation.cc:299] Negotiation successful
0928 06:42:32.833037 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260928 06:42:32.833297 1380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:32.825179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.66:47179)
0928 06:42:32.825327 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:32.825365 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:32.825846 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:32.826076 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:32.826079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:32.826704 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:42:32.826899 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:32.826905 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.827613 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.827615 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:32.828494 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:32.828497 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.828886 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.828887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:32.829010 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:32.829450 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:42:32.829458 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:42:32.830762 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:42:32.832098 (+ 1336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:32.832104 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:32.832107 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:32.832360 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:32.832514 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:32.832514 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:32.832515 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:32.832603 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:32.832772 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:42:32.832775 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:42:32.832883 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 06:42:32.832958 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 06:42:32.833071 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":76,"threads_started":1}
I20260928 06:42:32.833666 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 1244
I20260928 06:42:32.833776 1378 heartbeater.cc:347] Connected to a master server at 127.18.211.126:33269
I20260928 06:42:32.833861 1378 heartbeater.cc:464] Registering TS with master...
I20260928 06:42:32.833948 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.67:43049
--local_ip_for_outbound_sockets=127.18.211.67
--tserver_master_addrs=127.18.211.126:33269
--webserver_port=38399
--webserver_interface=127.18.211.67
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:42:32.834035 1378 heartbeater.cc:511] Master 127.18.211.126:33269 requested a full tablet report, sending...
I20260928 06:42:32.834411 1050 ts_manager.cc:194] Registered new tserver with Master: a34c9d02780440f68a330a63df8da0a9 (127.18.211.66:36929)
I20260928 06:42:32.834884 1050 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.66@KRBTEST.COM'} at 127.18.211.66:47179
W20260928 06:42:32.916909 1381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:42:32.917104 1381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:42:32.917124 1381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:42:32.918236 1381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:42:32.918279 1381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:42:32.918304 1381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.67
I20260928 06:42:32.919768 1381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.67:43049
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.211.67
--webserver_port=38399
--tserver_master_addrs=127.18.211.126:33269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.67
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:42:32.919953 1381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:42:32.920126 1381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:42:32.924875 1386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:32.924916 1389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:42:32.924990 1381 server_base.cc:1034] running on GCE node
W20260928 06:42:32.924896 1387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:42:32.925293 1381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:42:32.925508 1381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:42:32.926662 1381 hybrid_clock.cc:648] HybridClock initialized: now 1790577752926643 us; error 35 us; skew 500 ppm
Sep 28 06:42:32 dist-test-slave-b28r krb5kdc[469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577752, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:42:32.929171 1381 init.cc:377] Logged in from keytab as kudu/127.18.211.67@KRBTEST.COM (short username kudu)
I20260928 06:42:32.929541 1381 webserver.cc:460] Webserver started at http://127.18.211.67:38399/ using document root <none> and password file <none>
I20260928 06:42:32.929733 1381 fs_manager.cc:359] Metadata directory not provided
I20260928 06:42:32.929786 1381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:42:32.931020 1381 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:42:32.931588 1396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:32.931766 1381 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:32.931854 1381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/wal
uuid: "47100e01c7b84ed6b99fbebe03ed7e12"
format_stamp: "Formatted at 2026-09-28 06:42:32 on dist-test-slave-b28r"
server_key: "849b14553b2f722f4df6629ecebcf89e"
server_key_iv: "39d5a664f7d14d54167aceb03539f71b"
server_key_version: "encryptionkey@0"
I20260928 06:42:32.932196 1381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:42:32.939565 1381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:42:32.941249 1381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:42:32.941366 1381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:42:32.941553 1381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:42:32.941826 1381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:42:32.941859 1381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:32.941879 1381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:42:32.941900 1381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:32.947358 1381 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.67:43049
I20260928 06:42:32.947425 1509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.67:43049 every 8 connection(s)
I20260928 06:42:32.947713 1381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
I20260928 06:42:32.948705 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 1381
Sep 28 06:42:32 dist-test-slave-b28r krb5kdc[469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577752, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:42:32.956472 1514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:32.949061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58024 (local address 127.18.211.126:33269)
0928 06:42:32.949249 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:32.949250 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:32.949393 (+ 143us) server_negotiation.cc:407] Connection header received
0928 06:42:32.949542 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:32.949542 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:32.949548 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:32.949587 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:32.950390 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.950893 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.951651 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.951882 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.951936 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:32.953142 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:32.953143 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:32.953143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:32.953152 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:32.954863 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:32.955241 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:32.955242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:32.955243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:32.955308 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:32.955490 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:32.955491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:32.955491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:32.955614 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:32.956312 (+ 698us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:32.956345 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 06:42:32.956403 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":52,"threads_started":1}
I20260928 06:42:32.957473 1242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:32.948465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:33785 (local address 127.18.211.126:33269)
0928 06:42:32.948550 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:32.948553 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:32.948718 (+ 165us) server_negotiation.cc:407] Connection header received
0928 06:42:32.949449 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:32.949450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:32.949459 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:32.949506 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:32.950637 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.951176 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.952177 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.952368 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.952422 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:32.954847 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:32.954848 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:32.954849 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:32.954855 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:32.956134 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:32.956520 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:32.956520 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:32.956521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:32.956576 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:32.956859 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:32.956860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:32.956860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:32.956990 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:32.957049 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:32.957354 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 06:42:32.957414 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 06:42:32.957626 1512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:32.948398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.67:33785)
0928 06:42:32.948574 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:32.948618 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:32.949291 (+ 673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:32.949577 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:32.949581 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:32.950314 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:42:32.950542 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:32.950548 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.951259 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.951260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:32.952089 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:32.952091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.952451 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.952452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:32.952580 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:32.953069 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:42:32.953078 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:42:32.954754 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:42:32.956220 (+ 1466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:32.956226 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:32.956228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:32.956447 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:32.956652 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:32.956653 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:32.956653 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:32.956794 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:32.957053 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:42:32.957059 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:42:32.957211 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 06:42:32.957321 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 06:42:32.957444 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":76,"threads_started":1}
I20260928 06:42:32.958042 1510 heartbeater.cc:347] Connected to a master server at 127.18.211.126:33269
I20260928 06:42:32.958122 1510 heartbeater.cc:464] Registering TS with master...
I20260928 06:42:32.958266 1510 heartbeater.cc:511] Master 127.18.211.126:33269 requested a full tablet report, sending...
I20260928 06:42:32.958552 1050 ts_manager.cc:194] Registered new tserver with Master: 47100e01c7b84ed6b99fbebe03ed7e12 (127.18.211.67:43049)
I20260928 06:42:32.958900 1050 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.67@KRBTEST.COM'} at 127.18.211.67:33785
I20260928 06:42:32.968164 19277 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:42:32.970836 1242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:32.968515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58032 (local address 127.18.211.126:33269)
0928 06:42:32.968566 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:32.968568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:32.968698 (+ 130us) server_negotiation.cc:407] Connection header received
0928 06:42:32.968756 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:32.968757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:32.968789 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:32.968846 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:42:32.968936 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.969429 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.970177 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.970374 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.970434 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:32.970560 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:42:32.970658 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:42:32.970721 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:32.970746 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 06:42:32.970762 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:42:32.971199 1050 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:58032:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:42:32.972751 1050 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:42:32.974742 1173 tablet_service.cc:1467] Processing CreateTablet for tablet 32fb705c8d2940018e362a7610f676f7 (DEFAULT_TABLE table=test-table [id=1b0fef2c6f524704862cc140e35c58e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:42:32.974944 1173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32fb705c8d2940018e362a7610f676f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:42:32.976596 1130 tablet_bootstrap.cc:492] T 32fb705c8d2940018e362a7610f676f7 P a8a75496766348418bca6d243610c5dc: Bootstrap starting.
I20260928 06:42:32.976918 1130 tablet_bootstrap.cc:654] T 32fb705c8d2940018e362a7610f676f7 P a8a75496766348418bca6d243610c5dc: Neither blocks nor log segments found. Creating new log.
I20260928 06:42:32.977540 1130 tablet_bootstrap.cc:492] T 32fb705c8d2940018e362a7610f676f7 P a8a75496766348418bca6d243610c5dc: No bootstrap required, opened a new log
I20260928 06:42:32.977587 1130 ts_tablet_manager.cc:1397] T 32fb705c8d2940018e362a7610f676f7 P a8a75496766348418bca6d243610c5dc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260928 06:42:32.977898 1130 raft_consensus.cc:348] T 32fb705c8d2940018e362a7610f676f7 P a8a75496766348418bca6d243610c5dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47100e01c7b84ed6b99fbebe03ed7e12" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 43049 } } peers { permanent_uuid: "a34c9d02780440f68a330a63df8da0a9" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36929 } } peers { permanent_uuid: "a8a75496766348418bca6d243610c5dc" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 42529 } }
I20260928 06:42:32.977960 1130 raft_consensus.cc:374] T 32fb705c8d2940018e362a7610f676f7 P a8a75496766348418bca6d243610c5dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:42:32.978003 1130 raft_consensus.cc:729] T 32fb705c8d2940018e362a7610f676f7 P a8a75496766348418bca6d243610c5dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8a75496766348418bca6d243610c5dc, State: Initialized, Role: FOLLOWER
I20260928 06:42:32.978055 1130 consensus_queue.cc:260] T 32fb705c8d2940018e362a7610f676f7 P a8a75496766348418bca6d243610c5dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47100e01c7b84ed6b99fbebe03ed7e12" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 43049 } } peers { permanent_uuid: "a34c9d02780440f68a330a63df8da0a9" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36929 } } peers { permanent_uuid: "a8a75496766348418bca6d243610c5dc" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 42529 } }
I20260928 06:42:32.978142 1130 ts_tablet_manager.cc:1428] T 32fb705c8d2940018e362a7610f676f7 P a8a75496766348418bca6d243610c5dc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 06:42:32.978276 1246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:32.974416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:43049 (local address 127.0.0.1:35772)
0928 06:42:32.974466 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:32.974477 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:32.974515 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:32.975009 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:32.975010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:32.975012 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:42:32.975055 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:32.975056 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.976144 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.976145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:32.977099 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:32.977102 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.977976 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.977977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:32.978052 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:32.978074 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 06:42:32.978143 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 06:42:32.978208 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 06:42:32.978338 1517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:32.974638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35772 (local address 127.18.211.67:43049)
0928 06:42:32.974854 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:32.974859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:32.974871 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:42:32.974909 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:32.974911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:32.974927 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:32.975004 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:42:32.975177 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.976058 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.977169 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.977894 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.977961 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:32.978015 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:32.978206 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 06:42:32.978248 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":130,"threads_started":1}
I20260928 06:42:32.978775 1516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:32.974465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:36929 (local address 127.0.0.1:41822)
0928 06:42:32.974605 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:32.974613 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:32.974646 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:32.975359 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:32.975360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:32.975362 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:42:32.975400 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:32.975401 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.976586 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.976587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:32.977546 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:32.977549 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.978529 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.978530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:32.978592 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:32.978597 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:42:32.978653 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 06:42:32.978694 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":100,"threads_started":1}
I20260928 06:42:32.979187 1444 tablet_service.cc:1467] Processing CreateTablet for tablet 32fb705c8d2940018e362a7610f676f7 (DEFAULT_TABLE table=test-table [id=1b0fef2c6f524704862cc140e35c58e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:42:32.979543 1444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32fb705c8d2940018e362a7610f676f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:42:32.982162 1520 tablet_bootstrap.cc:492] T 32fb705c8d2940018e362a7610f676f7 P 47100e01c7b84ed6b99fbebe03ed7e12: Bootstrap starting.
I20260928 06:42:32.982232 1518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:32.974938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41822 (local address 127.18.211.66:36929)
0928 06:42:32.975197 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:32.975202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:32.975213 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:42:32.975251 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:32.975254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:32.975268 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:32.975347 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:42:32.975460 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.976467 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.977639 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:32.978381 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:32.978445 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:32.978498 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:32.978690 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 06:42:32.982147 (+ 3457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":98,"threads_started":1}
I20260928 06:42:32.982838 1520 tablet_bootstrap.cc:654] T 32fb705c8d2940018e362a7610f676f7 P 47100e01c7b84ed6b99fbebe03ed7e12: Neither blocks nor log segments found. Creating new log.
I20260928 06:42:32.982931 1312 tablet_service.cc:1467] Processing CreateTablet for tablet 32fb705c8d2940018e362a7610f676f7 (DEFAULT_TABLE table=test-table [id=1b0fef2c6f524704862cc140e35c58e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:42:32.983314 1312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32fb705c8d2940018e362a7610f676f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:42:32.983481 1520 log.cc:824] T 32fb705c8d2940018e362a7610f676f7 P 47100e01c7b84ed6b99fbebe03ed7e12: Log is configured to *not* fsync() on all Append() calls
I20260928 06:42:32.984211 1520 tablet_bootstrap.cc:492] T 32fb705c8d2940018e362a7610f676f7 P 47100e01c7b84ed6b99fbebe03ed7e12: No bootstrap required, opened a new log
I20260928 06:42:32.984287 1520 ts_tablet_manager.cc:1397] T 32fb705c8d2940018e362a7610f676f7 P 47100e01c7b84ed6b99fbebe03ed7e12: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:42:32.986136 1520 raft_consensus.cc:348] T 32fb705c8d2940018e362a7610f676f7 P 47100e01c7b84ed6b99fbebe03ed7e12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47100e01c7b84ed6b99fbebe03ed7e12" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 43049 } } peers { permanent_uuid: "a34c9d02780440f68a330a63df8da0a9" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36929 } } peers { permanent_uuid: "a8a75496766348418bca6d243610c5dc" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 42529 } }
I20260928 06:42:32.986320 1520 raft_consensus.cc:374] T 32fb705c8d2940018e362a7610f676f7 P 47100e01c7b84ed6b99fbebe03ed7e12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:42:32.986356 1520 raft_consensus.cc:729] T 32fb705c8d2940018e362a7610f676f7 P 47100e01c7b84ed6b99fbebe03ed7e12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47100e01c7b84ed6b99fbebe03ed7e12, State: Initialized, Role: FOLLOWER
I20260928 06:42:32.986478 1520 consensus_queue.cc:260] T 32fb705c8d2940018e362a7610f676f7 P 47100e01c7b84ed6b99fbebe03ed7e12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47100e01c7b84ed6b99fbebe03ed7e12" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 43049 } } peers { permanent_uuid: "a34c9d02780440f68a330a63df8da0a9" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36929 } } peers { permanent_uuid: "a8a75496766348418bca6d243610c5dc" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 42529 } }
I20260928 06:42:32.986742 1520 ts_tablet_manager.cc:1428] T 32fb705c8d2940018e362a7610f676f7 P 47100e01c7b84ed6b99fbebe03ed7e12: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:42:32.986738 1522 tablet_bootstrap.cc:492] T 32fb705c8d2940018e362a7610f676f7 P a34c9d02780440f68a330a63df8da0a9: Bootstrap starting.
I20260928 06:42:32.986770 1510 heartbeater.cc:503] Master 127.18.211.126:33269 was elected leader, sending a full tablet report...
I20260928 06:42:32.987375 1522 tablet_bootstrap.cc:654] T 32fb705c8d2940018e362a7610f676f7 P a34c9d02780440f68a330a63df8da0a9: Neither blocks nor log segments found. Creating new log.
I20260928 06:42:32.987887 1522 log.cc:824] T 32fb705c8d2940018e362a7610f676f7 P a34c9d02780440f68a330a63df8da0a9: Log is configured to *not* fsync() on all Append() calls
I20260928 06:42:32.988544 1522 tablet_bootstrap.cc:492] T 32fb705c8d2940018e362a7610f676f7 P a34c9d02780440f68a330a63df8da0a9: No bootstrap required, opened a new log
I20260928 06:42:32.988603 1522 ts_tablet_manager.cc:1397] T 32fb705c8d2940018e362a7610f676f7 P a34c9d02780440f68a330a63df8da0a9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 06:42:32.989887 1522 raft_consensus.cc:348] T 32fb705c8d2940018e362a7610f676f7 P a34c9d02780440f68a330a63df8da0a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47100e01c7b84ed6b99fbebe03ed7e12" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 43049 } } peers { permanent_uuid: "a34c9d02780440f68a330a63df8da0a9" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36929 } } peers { permanent_uuid: "a8a75496766348418bca6d243610c5dc" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 42529 } }
I20260928 06:42:32.990008 1522 raft_consensus.cc:374] T 32fb705c8d2940018e362a7610f676f7 P a34c9d02780440f68a330a63df8da0a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:42:32.990041 1522 raft_consensus.cc:729] T 32fb705c8d2940018e362a7610f676f7 P a34c9d02780440f68a330a63df8da0a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a34c9d02780440f68a330a63df8da0a9, State: Initialized, Role: FOLLOWER
I20260928 06:42:32.990145 1522 consensus_queue.cc:260] T 32fb705c8d2940018e362a7610f676f7 P a34c9d02780440f68a330a63df8da0a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47100e01c7b84ed6b99fbebe03ed7e12" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 43049 } } peers { permanent_uuid: "a34c9d02780440f68a330a63df8da0a9" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36929 } } peers { permanent_uuid: "a8a75496766348418bca6d243610c5dc" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 42529 } }
I20260928 06:42:32.990430 1522 ts_tablet_manager.cc:1428] T 32fb705c8d2940018e362a7610f676f7 P a34c9d02780440f68a330a63df8da0a9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:42:32.990500 1378 heartbeater.cc:503] Master 127.18.211.126:33269 was elected leader, sending a full tablet report...
W20260928 06:42:33.075147 1379 tablet.cc:2369] T 32fb705c8d2940018e362a7610f676f7 P a34c9d02780440f68a330a63df8da0a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:42:33.089529 1245 raft_consensus.cc:482] T 32fb705c8d2940018e362a7610f676f7 P a8a75496766348418bca6d243610c5dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:42:33.089681 1245 raft_consensus.cc:504] T 32fb705c8d2940018e362a7610f676f7 P a8a75496766348418bca6d243610c5dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47100e01c7b84ed6b99fbebe03ed7e12" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 43049 } } peers { permanent_uuid: "a34c9d02780440f68a330a63df8da0a9" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36929 } } peers { permanent_uuid: "a8a75496766348418bca6d243610c5dc" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 42529 } }
I20260928 06:42:33.090019 1245 leader_election.cc:288] T 32fb705c8d2940018e362a7610f676f7 P a8a75496766348418bca6d243610c5dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47100e01c7b84ed6b99fbebe03ed7e12 (127.18.211.67:43049), a34c9d02780440f68a330a63df8da0a9 (127.18.211.66:36929)
I20260928 06:42:33.092922 1517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:33.090290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:41565 (local address 127.18.211.67:43049)
0928 06:42:33.090352 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:33.090354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:33.090366 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:42:33.090426 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:33.090427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:33.090437 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:33.090493 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:42:33.090671 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:33.091222 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:33.092176 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:33.092566 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:33.092619 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:33.092626 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:33.092812 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 06:42:33.092850 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:42:33.092949 1518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:33.090412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:38985 (local address 127.18.211.66:36929)
0928 06:42:33.090459 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:33.090460 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:33.090467 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:42:33.090497 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:33.090498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:33.090506 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:33.090567 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:42:33.090726 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:33.091213 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:33.092220 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:33.092598 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:33.092667 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:33.092675 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:33.092846 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 06:42:33.092889 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 06:42:33.092945 1241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:33.090172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:43049 (local address 127.18.211.65:41565)
0928 06:42:33.090265 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:33.090277 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:33.090353 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:33.090502 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:33.090503 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:33.090505 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:42:33.090583 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:33.090585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:33.091304 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:33.091305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:33.092097 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:33.092099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:33.092635 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:33.092636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:33.092726 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:33.092748 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 06:42:33.092788 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 06:42:33.092826 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 06:42:33.093096 1526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:33.090231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:36929 (local address 127.18.211.65:38985)
0928 06:42:33.090387 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:33.090397 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:33.090447 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:33.090578 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:33.090579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:33.090581 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:42:33.090654 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:33.090656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:33.091288 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:33.091290 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:33.092123 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:33.092126 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:33.092685 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:33.092686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:33.092738 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:33.092771 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 06:42:33.092831 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 06:42:33.092867 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":86,"threads_started":1}
I20260928 06:42:33.093290 1464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fb705c8d2940018e362a7610f676f7" candidate_uuid: "a8a75496766348418bca6d243610c5dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47100e01c7b84ed6b99fbebe03ed7e12" is_pre_election: true
I20260928 06:42:33.093463 1464 raft_consensus.cc:2454] T 32fb705c8d2940018e362a7610f676f7 P 47100e01c7b84ed6b99fbebe03ed7e12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8a75496766348418bca6d243610c5dc in term 0.
I20260928 06:42:33.093420 1332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fb705c8d2940018e362a7610f676f7" candidate_uuid: "a8a75496766348418bca6d243610c5dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a34c9d02780440f68a330a63df8da0a9" is_pre_election: true
I20260928 06:42:33.093552 1332 raft_consensus.cc:2454] T 32fb705c8d2940018e362a7610f676f7 P a34c9d02780440f68a330a63df8da0a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8a75496766348418bca6d243610c5dc in term 0.
I20260928 06:42:33.093713 1126 leader_election.cc:302] T 32fb705c8d2940018e362a7610f676f7 P a8a75496766348418bca6d243610c5dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47100e01c7b84ed6b99fbebe03ed7e12, a8a75496766348418bca6d243610c5dc; no voters:
I20260928 06:42:33.093868 1245 raft_consensus.cc:2781] T 32fb705c8d2940018e362a7610f676f7 P a8a75496766348418bca6d243610c5dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:42:33.093922 1245 raft_consensus.cc:482] T 32fb705c8d2940018e362a7610f676f7 P a8a75496766348418bca6d243610c5dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:42:33.093937 1245 raft_consensus.cc:3037] T 32fb705c8d2940018e362a7610f676f7 P a8a75496766348418bca6d243610c5dc [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:42:33.094574 1245 raft_consensus.cc:504] T 32fb705c8d2940018e362a7610f676f7 P a8a75496766348418bca6d243610c5dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47100e01c7b84ed6b99fbebe03ed7e12" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 43049 } } peers { permanent_uuid: "a34c9d02780440f68a330a63df8da0a9" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36929 } } peers { permanent_uuid: "a8a75496766348418bca6d243610c5dc" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 42529 } }
I20260928 06:42:33.094705 1245 leader_election.cc:288] T 32fb705c8d2940018e362a7610f676f7 P a8a75496766348418bca6d243610c5dc [CANDIDATE]: Term 1 election: Requested vote from peers 47100e01c7b84ed6b99fbebe03ed7e12 (127.18.211.67:43049), a34c9d02780440f68a330a63df8da0a9 (127.18.211.66:36929)
I20260928 06:42:33.094838 1464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fb705c8d2940018e362a7610f676f7" candidate_uuid: "a8a75496766348418bca6d243610c5dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47100e01c7b84ed6b99fbebe03ed7e12"
I20260928 06:42:33.094861 1332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fb705c8d2940018e362a7610f676f7" candidate_uuid: "a8a75496766348418bca6d243610c5dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a34c9d02780440f68a330a63df8da0a9"
I20260928 06:42:33.094920 1464 raft_consensus.cc:3037] T 32fb705c8d2940018e362a7610f676f7 P 47100e01c7b84ed6b99fbebe03ed7e12 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:42:33.094933 1332 raft_consensus.cc:3037] T 32fb705c8d2940018e362a7610f676f7 P a34c9d02780440f68a330a63df8da0a9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:42:33.095667 1464 raft_consensus.cc:2454] T 32fb705c8d2940018e362a7610f676f7 P 47100e01c7b84ed6b99fbebe03ed7e12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8a75496766348418bca6d243610c5dc in term 1.
I20260928 06:42:33.095705 1332 raft_consensus.cc:2454] T 32fb705c8d2940018e362a7610f676f7 P a34c9d02780440f68a330a63df8da0a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8a75496766348418bca6d243610c5dc in term 1.
I20260928 06:42:33.095824 1126 leader_election.cc:302] T 32fb705c8d2940018e362a7610f676f7 P a8a75496766348418bca6d243610c5dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47100e01c7b84ed6b99fbebe03ed7e12, a8a75496766348418bca6d243610c5dc; no voters:
I20260928 06:42:33.095942 1245 raft_consensus.cc:2781] T 32fb705c8d2940018e362a7610f676f7 P a8a75496766348418bca6d243610c5dc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:42:33.096104 1245 raft_consensus.cc:686] T 32fb705c8d2940018e362a7610f676f7 P a8a75496766348418bca6d243610c5dc [term 1 LEADER]: Becoming Leader. State: Replica: a8a75496766348418bca6d243610c5dc, State: Running, Role: LEADER
I20260928 06:42:33.096185 1245 consensus_queue.cc:237] T 32fb705c8d2940018e362a7610f676f7 P a8a75496766348418bca6d243610c5dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47100e01c7b84ed6b99fbebe03ed7e12" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 43049 } } peers { permanent_uuid: "a34c9d02780440f68a330a63df8da0a9" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36929 } } peers { permanent_uuid: "a8a75496766348418bca6d243610c5dc" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 42529 } }
I20260928 06:42:33.096923 1050 catalog_manager.cc:5495] T 32fb705c8d2940018e362a7610f676f7 P a8a75496766348418bca6d243610c5dc reported cstate change: term changed from 0 to 1, leader changed from <none> to a8a75496766348418bca6d243610c5dc (127.18.211.65). New cstate: current_term: 1 leader_uuid: "a8a75496766348418bca6d243610c5dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47100e01c7b84ed6b99fbebe03ed7e12" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 43049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a34c9d02780440f68a330a63df8da0a9" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8a75496766348418bca6d243610c5dc" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 42529 } health_report { overall_health: HEALTHY } } }
I20260928 06:42:33.123560 1247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:33.121104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50410 (local address 127.18.211.65:42529)
0928 06:42:33.121163 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:33.121165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:33.121173 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:42:33.121221 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:33.121222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:33.121242 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:33.121302 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:42:33.121436 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:33.121989 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:33.122723 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:33.122992 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:33.123048 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:33.123232 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:42:33.123350 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:42:33.123412 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:33.123460 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 06:42:33.123482 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 06:42:33.125186 1464 raft_consensus.cc:1261] T 32fb705c8d2940018e362a7610f676f7 P 47100e01c7b84ed6b99fbebe03ed7e12 [term 1 FOLLOWER]: Refusing update from remote peer a8a75496766348418bca6d243610c5dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:42:33.125186 1332 raft_consensus.cc:1261] T 32fb705c8d2940018e362a7610f676f7 P a34c9d02780440f68a330a63df8da0a9 [term 1 FOLLOWER]: Refusing update from remote peer a8a75496766348418bca6d243610c5dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:42:33.125522 1245 consensus_queue.cc:1035] T 32fb705c8d2940018e362a7610f676f7 P a8a75496766348418bca6d243610c5dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "47100e01c7b84ed6b99fbebe03ed7e12" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 43049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:42:33.125675 1527 consensus_queue.cc:1035] T 32fb705c8d2940018e362a7610f676f7 P a8a75496766348418bca6d243610c5dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a34c9d02780440f68a330a63df8da0a9" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:42:33.127831 1530 mvcc.cc:205] Tried to move back new op lower bound from 7334206476797775872 to 7334206476682452992. Current Snapshot: MvccSnapshot[applied={T|T < 7334206476797775872}]
I20260928 06:42:33.128280 1532 mvcc.cc:205] Tried to move back new op lower bound from 7334206476797775872 to 7334206476682452992. Current Snapshot: MvccSnapshot[applied={T|T < 7334206476797775872}]
I20260928 06:42:33.130070 1049 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:58032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:42:33.130165 1049 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:58032:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:42:33.130934 1049 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e: Sending DeleteTablet for 3 replicas of tablet 32fb705c8d2940018e362a7610f676f7
I20260928 06:42:33.131376 1444 tablet_service.cc:1514] Processing DeleteTablet for tablet 32fb705c8d2940018e362a7610f676f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:42:33 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:35772
I20260928 06:42:33.131494 1173 tablet_service.cc:1514] Processing DeleteTablet for tablet 32fb705c8d2940018e362a7610f676f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:42:33 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:50406
I20260928 06:42:33.131510 1312 tablet_service.cc:1514] Processing DeleteTablet for tablet 32fb705c8d2940018e362a7610f676f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:42:33 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:41822
I20260928 06:42:33.131580 1248 tablet_replica.cc:330] T 32fb705c8d2940018e362a7610f676f7 P a8a75496766348418bca6d243610c5dc: stopping tablet replica
I20260928 06:42:33.131630 1248 raft_consensus.cc:2229] T 32fb705c8d2940018e362a7610f676f7 P a8a75496766348418bca6d243610c5dc [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:42:33.131712 1248 raft_consensus.cc:2258] T 32fb705c8d2940018e362a7610f676f7 P a8a75496766348418bca6d243610c5dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:42:33.131733 1541 tablet_replica.cc:330] T 32fb705c8d2940018e362a7610f676f7 P a34c9d02780440f68a330a63df8da0a9: stopping tablet replica
I20260928 06:42:33.131904 1540 tablet_replica.cc:330] T 32fb705c8d2940018e362a7610f676f7 P 47100e01c7b84ed6b99fbebe03ed7e12: stopping tablet replica
I20260928 06:42:33.131915 1541 raft_consensus.cc:2229] T 32fb705c8d2940018e362a7610f676f7 P a34c9d02780440f68a330a63df8da0a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:42:33.132009 1540 raft_consensus.cc:2229] T 32fb705c8d2940018e362a7610f676f7 P 47100e01c7b84ed6b99fbebe03ed7e12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:42:33.132089 1248 ts_tablet_manager.cc:1905] T 32fb705c8d2940018e362a7610f676f7 P a8a75496766348418bca6d243610c5dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:42:33.132113 1540 raft_consensus.cc:2258] T 32fb705c8d2940018e362a7610f676f7 P 47100e01c7b84ed6b99fbebe03ed7e12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:42:33.132191 1541 raft_consensus.cc:2258] T 32fb705c8d2940018e362a7610f676f7 P a34c9d02780440f68a330a63df8da0a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:42:33.132504 1540 ts_tablet_manager.cc:1905] T 32fb705c8d2940018e362a7610f676f7 P 47100e01c7b84ed6b99fbebe03ed7e12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:42:33.132735 1541 ts_tablet_manager.cc:1905] T 32fb705c8d2940018e362a7610f676f7 P a34c9d02780440f68a330a63df8da0a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:42:33.133589 1248 ts_tablet_manager.cc:1918] T 32fb705c8d2940018e362a7610f676f7 P a8a75496766348418bca6d243610c5dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:42:33.133647 1248 log.cc:1196] T 32fb705c8d2940018e362a7610f676f7 P a8a75496766348418bca6d243610c5dc: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/wal/wals/32fb705c8d2940018e362a7610f676f7
I20260928 06:42:33.133873 1248 ts_tablet_manager.cc:1939] T 32fb705c8d2940018e362a7610f676f7 P a8a75496766348418bca6d243610c5dc: Deleting consensus metadata
I20260928 06:42:33.133987 1540 ts_tablet_manager.cc:1918] T 32fb705c8d2940018e362a7610f676f7 P 47100e01c7b84ed6b99fbebe03ed7e12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:42:33.134054 1540 log.cc:1196] T 32fb705c8d2940018e362a7610f676f7 P 47100e01c7b84ed6b99fbebe03ed7e12: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/wal/wals/32fb705c8d2940018e362a7610f676f7
I20260928 06:42:33.134160 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 1088
I20260928 06:42:33.134327 1540 ts_tablet_manager.cc:1939] T 32fb705c8d2940018e362a7610f676f7 P 47100e01c7b84ed6b99fbebe03ed7e12: Deleting consensus metadata
I20260928 06:42:33.134974 1036 catalog_manager.cc:4841] TS 47100e01c7b84ed6b99fbebe03ed7e12 (127.18.211.67:43049): tablet 32fb705c8d2940018e362a7610f676f7 (table test-table [id=1b0fef2c6f524704862cc140e35c58e2]) successfully deleted
I20260928 06:42:33.135385 1541 ts_tablet_manager.cc:1918] T 32fb705c8d2940018e362a7610f676f7 P a34c9d02780440f68a330a63df8da0a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:42:33.135502 1541 log.cc:1196] T 32fb705c8d2940018e362a7610f676f7 P a34c9d02780440f68a330a63df8da0a9: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/wal/wals/32fb705c8d2940018e362a7610f676f7
I20260928 06:42:33.135815 1541 ts_tablet_manager.cc:1939] T 32fb705c8d2940018e362a7610f676f7 P a34c9d02780440f68a330a63df8da0a9: Deleting consensus metadata
I20260928 06:42:33.136270 1034 catalog_manager.cc:4841] TS a34c9d02780440f68a330a63df8da0a9 (127.18.211.66:36929): tablet 32fb705c8d2940018e362a7610f676f7 (table test-table [id=1b0fef2c6f524704862cc140e35c58e2]) successfully deleted
W20260928 06:42:33.138743 1036 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.211.65:42529 (error 108)
I20260928 06:42:33.139102 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 1244
I20260928 06:42:33.139222 1516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:33.139103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:42529 (local address 127.0.0.1:50426)
0928 06:42:33.139147 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:33.139189 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.65:42529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260928 06:42:33.139710 1036 catalog_manager.cc:4584] TS a8a75496766348418bca6d243610c5dc (127.18.211.65:42529): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 32fb705c8d2940018e362a7610f676f7: Network error: Client connection negotiation failed: client connection to 127.18.211.65:42529: connect: Connection refused (error 111)
I20260928 06:42:33.144045 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 1381
I20260928 06:42:33.147806 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 1017
I20260928 06:42:33.152730 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:33269
--webserver_interface=127.18.211.126
--webserver_port=34409
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:33269
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:42:33.232291 1543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:42:33.232508 1543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:42:33.232529 1543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:42:33.233596 1543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:42:33.233630 1543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:42:33.233644 1543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:42:33.233654 1543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:42:33.233664 1543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:42:33.235046 1543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:33269
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:33269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=34409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:42:33.235272 1543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:42:33.235438 1543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:42:33.240028 1551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:33.240008 1548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:42:33.240042 1543 server_base.cc:1034] running on GCE node
W20260928 06:42:33.240000 1549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:42:33.240584 1543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:42:33.240866 1543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:42:33.242035 1543 hybrid_clock.cc:648] HybridClock initialized: now 1790577753242018 us; error 39 us; skew 500 ppm
Sep 28 06:42:33 dist-test-slave-b28r krb5kdc[469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577753, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:42:33.244679 1543 init.cc:377] Logged in from keytab as kudu/127.18.211.126@KRBTEST.COM (short username kudu)
I20260928 06:42:33.245137 1543 webserver.cc:460] Webserver started at http://127.18.211.126:34409/ using document root <none> and password file <none>
I20260928 06:42:33.245345 1543 fs_manager.cc:359] Metadata directory not provided
I20260928 06:42:33.245388 1543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:42:33.246677 1543 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:42:33.247422 1558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:33.247666 1543 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:42:33.247766 1543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "98ea104104c0443bbfad48f3cd5f272e"
format_stamp: "Formatted at 2026-09-28 06:42:31 on dist-test-slave-b28r"
server_key: "56564d69f13ab234aa9b81c472c2b297"
server_key_iv: "9f14f35f2f6196e399d4e389b3273995"
server_key_version: "encryptionkey@0"
I20260928 06:42:33.248111 1543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:42:33.259817 1543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:42:33.261833 1543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:42:33.261955 1543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:42:33.265637 1543 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:33269
I20260928 06:42:33.265697 1612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:33269 every 8 connection(s)
I20260928 06:42:33.266010 1543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:42:33.267343 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 1543
I20260928 06:42:33.267614 1613 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:42:33.267634 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.65:42529
--local_ip_for_outbound_sockets=127.18.211.65
--tserver_master_addrs=127.18.211.126:33269
--webserver_port=46339
--webserver_interface=127.18.211.65
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:42:33.268311 1613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e: Bootstrap starting.
I20260928 06:42:33.276741 1613 log.cc:824] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e: Log is configured to *not* fsync() on all Append() calls
I20260928 06:42:33.279242 1613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:42:33.279510 1613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e: Bootstrap complete.
I20260928 06:42:33.281632 1613 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ea104104c0443bbfad48f3cd5f272e" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 33269 } }
I20260928 06:42:33.281859 1613 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98ea104104c0443bbfad48f3cd5f272e, State: Initialized, Role: FOLLOWER
I20260928 06:42:33.282011 1613 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ea104104c0443bbfad48f3cd5f272e" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 33269 } }
I20260928 06:42:33.282085 1613 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:42:33.282132 1613 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:42:33.282176 1613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [term 2 FOLLOWER]: Advancing to term 3
I20260928 06:42:33.283149 1613 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ea104104c0443bbfad48f3cd5f272e" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 33269 } }
I20260928 06:42:33.283252 1613 leader_election.cc:302] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98ea104104c0443bbfad48f3cd5f272e; no voters:
I20260928 06:42:33.283424 1613 leader_election.cc:288] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 06:42:33.283473 1617 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [term 3 FOLLOWER]: Leader election won for term 3
I20260928 06:42:33.283581 1617 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [term 3 LEADER]: Becoming Leader. State: Replica: 98ea104104c0443bbfad48f3cd5f272e, State: Running, Role: LEADER
I20260928 06:42:33.283682 1613 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:42:33.283665 1617 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ea104104c0443bbfad48f3cd5f272e" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 33269 } }
I20260928 06:42:33.283923 1619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98ea104104c0443bbfad48f3cd5f272e. Latest consensus state: current_term: 3 leader_uuid: "98ea104104c0443bbfad48f3cd5f272e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ea104104c0443bbfad48f3cd5f272e" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 33269 } } }
I20260928 06:42:33.284062 1619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [sys.catalog]: This master's current role is: LEADER
I20260928 06:42:33.283929 1618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "98ea104104c0443bbfad48f3cd5f272e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ea104104c0443bbfad48f3cd5f272e" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 33269 } } }
I20260928 06:42:33.284152 1618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ea104104c0443bbfad48f3cd5f272e [sys.catalog]: This master's current role is: LEADER
I20260928 06:42:33.284348 1624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:42:33.285197 1624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:42:33.285332 1624 catalog_manager.cc:1259] Loaded cluster ID: 548c21bb1d884ab1bacc55b42743b6d3
I20260928 06:42:33.285542 1624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 06:42:33.285754 1624 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f97412689ad google::LogMessage::Fail() at ??:0
@ 0x7f974126b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f97412684de google::LogMessage::Flush() at ??:0
@ 0x7f9741269a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f97491890f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f97491928b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f97431360d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f974312b954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9743f8f6db start_thread at ??:0
@ 0x7f973fd6661f clone at ??:0
W20260928 06:42:33.362074 1615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:42:33.362268 1615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:42:33.362290 1615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:42:33.363447 1615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:42:33.363485 1615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:42:33.363509 1615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.65
I20260928 06:42:33.364900 1615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.65:42529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.211.65
--webserver_port=46339
--tserver_master_addrs=127.18.211.126:33269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.65
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:42:33.365108 1615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:42:33.365324 1615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:42:33.369918 1649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:33.369933 1650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:33.370066 1652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:42:33.370321 1615 server_base.cc:1034] running on GCE node
I20260928 06:42:33.370538 1615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:42:33.370777 1615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:42:33.371937 1615 hybrid_clock.cc:648] HybridClock initialized: now 1790577753371923 us; error 27 us; skew 500 ppm
Sep 28 06:42:33 dist-test-slave-b28r krb5kdc[469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577753, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:42:33.374991 1615 init.cc:377] Logged in from keytab as kudu/127.18.211.65@KRBTEST.COM (short username kudu)
I20260928 06:42:33.375587 1615 webserver.cc:460] Webserver started at http://127.18.211.65:46339/ using document root <none> and password file <none>
I20260928 06:42:33.375846 1615 fs_manager.cc:359] Metadata directory not provided
I20260928 06:42:33.375910 1615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:42:33.377633 1615 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:42:33.378331 1659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:33.378508 1615 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:42:33.378587 1615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/wal
uuid: "a8a75496766348418bca6d243610c5dc"
format_stamp: "Formatted at 2026-09-28 06:42:31 on dist-test-slave-b28r"
server_key: "2938121db54c2b6505cee2d6a936f3d6"
server_key_iv: "0909d0eaf6362a26fda324f17f0aa3ce"
server_key_version: "encryptionkey@0"
I20260928 06:42:33.378952 1615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:42:33.393710 1615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:42:33.395810 1615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:42:33.395926 1615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:42:33.396147 1615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:42:33.396433 1615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:42:33.396466 1615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:33.396485 1615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:42:33.396497 1615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:33.402050 1615 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.65:42529
I20260928 06:42:33.402107 1774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.65:42529 every 8 connection(s)
I20260928 06:42:33.402439 1615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
I20260928 06:42:33.402585 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 1615
I20260928 06:42:33.402878 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.66:36929
--local_ip_for_outbound_sockets=127.18.211.66
--tserver_master_addrs=127.18.211.126:33269
--webserver_port=38643
--webserver_interface=127.18.211.66
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:42:33.493355 1777 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:42:33.403242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.65:49203)
0928 06:42:33.403438 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:33.403489 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:33.404104 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:33.492845 (+ 88741us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":96,"threads_started":1}
W20260928 06:42:33.493635 1775 heartbeater.cc:644] Failed to heartbeat to 127.18.211.126:33269 (0 consecutive failures): Network error: Failed to ping master at 127.18.211.126:33269: Client connection negotiation failed: client connection to 127.18.211.126:33269: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 06:42:33.494050 1777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:33.493903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.65:35559)
0928 06:42:33.493963 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:33.494016 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 06:42:33.494427 1777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:33.494343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.65:60701)
0928 06:42:33.494384 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:33.494408 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 06:42:33.494536 1775 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:42:33.512348 1778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:42:33.512558 1778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:42:33.512590 1778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:42:33.513715 1778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:42:33.513754 1778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:42:33.513788 1778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.66
I20260928 06:42:33.515345 1778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.66:36929
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.211.66
--webserver_port=38643
--tserver_master_addrs=127.18.211.126:33269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.66
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:42:33.515575 1778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:42:33.515779 1778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:42:33.520705 1778 server_base.cc:1034] running on GCE node
W20260928 06:42:33.520679 1783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:33.520689 1786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:33.520717 1784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:42:33.521200 1778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:42:33.521436 1778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:42:33.522588 1778 hybrid_clock.cc:648] HybridClock initialized: now 1790577753522573 us; error 32 us; skew 500 ppm
Sep 28 06:42:33 dist-test-slave-b28r krb5kdc[469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577753, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:42:33.525087 1778 init.cc:377] Logged in from keytab as kudu/127.18.211.66@KRBTEST.COM (short username kudu)
I20260928 06:42:33.525545 1778 webserver.cc:460] Webserver started at http://127.18.211.66:38643/ using document root <none> and password file <none>
I20260928 06:42:33.525748 1778 fs_manager.cc:359] Metadata directory not provided
I20260928 06:42:33.525799 1778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:42:33.527140 1778 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:42:33.527779 1793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:33.527935 1778 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:42:33.528030 1778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/wal
uuid: "a34c9d02780440f68a330a63df8da0a9"
format_stamp: "Formatted at 2026-09-28 06:42:32 on dist-test-slave-b28r"
server_key: "bbf315a9465a08108bd594d407748d18"
server_key_iv: "929f02d6a143257417e06b4f4b5e5f14"
server_key_version: "encryptionkey@0"
I20260928 06:42:33.528419 1778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:42:33.539341 1778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:42:33.541508 1778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:42:33.541656 1778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:42:33.541884 1778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:42:33.542200 1778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:42:33.542243 1778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:33.542272 1778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:42:33.542296 1778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:33.547772 1778 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.66:36929
I20260928 06:42:33.547839 1907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.66:36929 every 8 connection(s)
I20260928 06:42:33.548177 1778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
I20260928 06:42:33.548830 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 1778
I20260928 06:42:33.549103 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.67:43049
--local_ip_for_outbound_sockets=127.18.211.67
--tserver_master_addrs=127.18.211.126:33269
--webserver_port=38399
--webserver_interface=127.18.211.67
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:42:33.549271 1910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:33.548797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.66:38017)
0928 06:42:33.548973 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:33.549077 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":74,"threads_started":1}
W20260928 06:42:33.549683 1908 heartbeater.cc:644] Failed to heartbeat to 127.18.211.126:33269 (0 consecutive failures): Network error: Failed to ping master at 127.18.211.126:33269: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
I20260928 06:42:33.549968 1910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:33.549883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.66:48185)
0928 06:42:33.549916 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:33.549946 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260928 06:42:33.550244 1910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:33.550145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.66:56245)
0928 06:42:33.550208 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:33.550229 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 06:42:33.550344 1908 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:42:33.628635 1911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:42:33.628835 1911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:42:33.628857 1911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:42:33.629995 1911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:42:33.630040 1911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:42:33.630067 1911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.67
I20260928 06:42:33.631554 1911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.67:43049
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.211.67
--webserver_port=38399
--tserver_master_addrs=127.18.211.126:33269
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.67
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:42:33.631786 1911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:42:33.632009 1911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:42:33.636825 1911 server_base.cc:1034] running on GCE node
W20260928 06:42:33.636768 1919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:33.636759 1917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:33.636785 1916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:42:33.637158 1911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:42:33.637403 1911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:42:33.638556 1911 hybrid_clock.cc:648] HybridClock initialized: now 1790577753638544 us; error 32 us; skew 500 ppm
Sep 28 06:42:33 dist-test-slave-b28r krb5kdc[469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577753, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:42:33.640935 1911 init.cc:377] Logged in from keytab as kudu/127.18.211.67@KRBTEST.COM (short username kudu)
I20260928 06:42:33.641306 1911 webserver.cc:460] Webserver started at http://127.18.211.67:38399/ using document root <none> and password file <none>
I20260928 06:42:33.641494 1911 fs_manager.cc:359] Metadata directory not provided
I20260928 06:42:33.641532 1911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:42:33.642716 1911 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:42:33.643244 1926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:33.643407 1911 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:33.643489 1911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/wal
uuid: "47100e01c7b84ed6b99fbebe03ed7e12"
format_stamp: "Formatted at 2026-09-28 06:42:32 on dist-test-slave-b28r"
server_key: "849b14553b2f722f4df6629ecebcf89e"
server_key_iv: "39d5a664f7d14d54167aceb03539f71b"
server_key_version: "encryptionkey@0"
I20260928 06:42:33.643807 1911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:42:33.662175 1911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:42:33.663982 1911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:42:33.664098 1911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:42:33.664322 1911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:42:33.664613 1911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:42:33.664646 1911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:33.664676 1911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:42:33.664700 1911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:33.670053 1911 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.67:43049
I20260928 06:42:33.670120 2040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.67:43049 every 8 connection(s)
I20260928 06:42:33.670408 1911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
I20260928 06:42:33.671413 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:33.671021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.67:40401)
0928 06:42:33.671199 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:33.671281 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":103,"threads_started":1}
W20260928 06:42:33.671778 2041 heartbeater.cc:644] Failed to heartbeat to 127.18.211.126:33269 (0 consecutive failures): Network error: Failed to ping master at 127.18.211.126:33269: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
I20260928 06:42:33.672017 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:33.671936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.67:55993)
0928 06:42:33.671975 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:33.671999 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 06:42:33.672248 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:33.672177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.67:56781)
0928 06:42:33.672210 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:33.672229 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260928 06:42:33.672338 2041 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 06:42:33.674305 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 1911
I20260928 06:42:34.495183 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:34.494890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.65:59625)
0928 06:42:34.495034 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:34.495072 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":64,"threads_started":1}
I20260928 06:42:34.550869 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:34.550664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.66:60837)
0928 06:42:34.550780 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:34.550816 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260928 06:42:34.672900 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:34.672666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.67:55613)
0928 06:42:34.672806 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:34.672844 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":61,"threads_started":1}
I20260928 06:42:35.495932 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:35.495672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.65:44011)
0928 06:42:35.495832 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:35.495879 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":83,"threads_started":1}
I20260928 06:42:35.551579 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:35.551353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.66:40919)
0928 06:42:35.551493 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:35.551536 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":70,"threads_started":1}
I20260928 06:42:35.673592 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:35.673364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.67:57355)
0928 06:42:35.673507 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:35.673548 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":58,"threads_started":1}
I20260928 06:42:36.496681 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:36.496461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.65:34525)
0928 06:42:36.496590 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:36.496633 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
I20260928 06:42:36.552223 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:36.552013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.66:57727)
0928 06:42:36.552138 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:36.552176 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260928 06:42:36.674301 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:36.674046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.67:55541)
0928 06:42:36.674218 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:36.674259 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":61,"threads_started":1}
I20260928 06:42:37.497462 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:37.497217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.65:59043)
0928 06:42:37.497376 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:37.497413 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":72,"threads_started":1}
I20260928 06:42:37.552908 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:37.552686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.66:58855)
0928 06:42:37.552840 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:37.552869 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":96,"threads_started":1}
I20260928 06:42:37.674922 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:37.674685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.67:45509)
0928 06:42:37.674842 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:37.674879 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":96,"threads_started":1}
I20260928 06:42:38.498243 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:38.497979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.65:42067)
0928 06:42:38.498142 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:38.498185 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":66,"threads_started":1}
I20260928 06:42:38.553583 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:38.553364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.66:59151)
0928 06:42:38.553504 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:38.553540 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
I20260928 06:42:38.675556 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:38.675320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.67:37081)
0928 06:42:38.675473 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:38.675511 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":79,"threads_started":1}
I20260928 06:42:39.498890 2060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:39.498675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.65:46183)
0928 06:42:39.498803 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:39.498842 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":61,"threads_started":1}
I20260928 06:42:39.554334 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:39.554063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.66:41835)
0928 06:42:39.554218 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:39.554269 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":82,"threads_started":1}
I20260928 06:42:39.676254 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:39.676013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.67:44821)
0928 06:42:39.676163 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:39.676206 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":80,"threads_started":1}
I20260928 06:42:40.499600 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:40.499373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.65:59861)
0928 06:42:40.499515 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:40.499555 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":64,"threads_started":1}
I20260928 06:42:40.554999 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:40.554783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.66:51631)
0928 06:42:40.554913 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:40.554954 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":67,"threads_started":1}
I20260928 06:42:40.676906 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:40.676688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.67:60933)
0928 06:42:40.676821 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:40.676861 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":68,"threads_started":1}
I20260928 06:42:41.500325 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:41.500091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.65:43387)
0928 06:42:41.500234 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:41.500273 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260928 06:42:41.555680 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:41.555417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.66:46555)
0928 06:42:41.555586 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:41.555633 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":88,"threads_started":1}
I20260928 06:42:41.677589 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:41.677344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.67:38877)
0928 06:42:41.677502 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:41.677541 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":81,"threads_started":1}
I20260928 06:42:42.501076 2069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:42.500793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.65:36343)
0928 06:42:42.500975 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:42.501026 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":85,"threads_started":1}
I20260928 06:42:42.556398 2070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:42.556168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.66:56015)
0928 06:42:42.556310 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:42.556348 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":82,"threads_started":1}
I20260928 06:42:42.678258 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:42.678035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.67:60629)
0928 06:42:42.678165 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:42.678210 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":58,"threads_started":1}
I20260928 06:42:43.501860 2072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:43.501617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.65:58943)
0928 06:42:43.501774 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:43.501812 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":71,"threads_started":1}
I20260928 06:42:43.557049 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:43.556839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.66:53365)
0928 06:42:43.556966 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:43.557003 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20260928 06:42:43.678951 2074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:43.678723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.67:60277)
0928 06:42:43.678868 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:43.678904 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":81,"threads_started":1}
I20260928 06:42:44.502736 2075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:44.502446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.65:37225)
0928 06:42:44.502605 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:44.502683 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":69,"threads_started":1}
I20260928 06:42:44.557718 2076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:44.557500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.66:42863)
0928 06:42:44.557633 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:44.557666 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
I20260928 06:42:44.679637 2077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:44.679417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.67:40043)
0928 06:42:44.679550 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:44.679591 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":80,"threads_started":1}
I20260928 06:42:45.503437 2078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:45.503208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.65:57155)
0928 06:42:45.503352 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:45.503384 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":88,"threads_started":1}
I20260928 06:42:45.558408 2079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:45.558143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.66:37749)
0928 06:42:45.558321 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:45.558361 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":109,"threads_started":1}
I20260928 06:42:45.680285 2080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:45.680089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.67:54987)
0928 06:42:45.680199 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:45.680236 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
I20260928 06:42:46.504185 2081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:46.503949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.65:39551)
0928 06:42:46.504089 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:46.504132 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
I20260928 06:42:46.559182 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:46.558892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.66:45725)
0928 06:42:46.559041 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:46.559118 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":83,"threads_started":1}
I20260928 06:42:46.680960 2083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:46.680726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.67:54633)
0928 06:42:46.680874 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:46.680913 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":68,"threads_started":1}
I20260928 06:42:47.504990 2084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:47.504750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.65:44329)
0928 06:42:47.504897 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:47.504940 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":65,"threads_started":1}
I20260928 06:42:47.559829 2085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:47.559629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.66:52197)
0928 06:42:47.559747 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:47.559784 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
I20260928 06:42:47.681674 2086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:47.681427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.67:51647)
0928 06:42:47.681568 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:47.681609 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":62,"threads_started":1}
I20260928 06:42:48.505710 2087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:48.505507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.65:48737)
0928 06:42:48.505631 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:48.505665 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260928 06:42:48.560523 2088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:48.560321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.66:51673)
0928 06:42:48.560443 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:48.560477 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":67,"threads_started":1}
I20260928 06:42:48.682513 2089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:48.682165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33269 (local address 127.18.211.67:48107)
0928 06:42:48.682385 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:48.682418 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:33269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":135,"threads_started":1}
I20260928 06:42:48.682525 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 1615
I20260928 06:42:48.686655 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 1778
I20260928 06:42:48.690455 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 1911
2026-09-28T06:42:48Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18030 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:42:48 dist-test-slave-b28r krb5kdc[2092](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:42:48 dist-test-slave-b28r krb5kdc[2092](info): set up 2 sockets
Sep 28 06:42:48 dist-test-slave-b28r krb5kdc[2092](info): commencing operation
krb5kdc: starting...
W20260928 06:42:49.643965 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:42:49 dist-test-slave-b28r krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:42:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:42:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.211.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
I20260928 06:42:49.751691 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:35839
--webserver_interface=127.18.211.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:35839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:42:49.829388 2105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:42:49.829581 2105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:42:49.829609 2105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:42:49.830730 2105 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 06:42:49.830766 2105 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 06:42:49.830782 2105 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 06:42:49.830796 2105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:42:49.830811 2105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:42:49.830838 2105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:42:49.830862 2105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:42:49.830880 2105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:42:49.832360 2105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:35839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:35839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:42:49.832602 2105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:42:49.832806 2105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:42:49.837680 2110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:49.837682 2111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:42:49.837682 2105 server_base.cc:1034] running on GCE node
W20260928 06:42:49.837774 2113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:42:49.838192 2105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:42:49.838443 2105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:42:49.839581 2105 hybrid_clock.cc:648] HybridClock initialized: now 1790577769839567 us; error 30 us; skew 500 ppm
Sep 28 06:42:49 dist-test-slave-b28r krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577769, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:42:49.841914 2105 init.cc:377] Logged in from keytab as kudu/127.18.211.126@KRBTEST.COM (short username kudu)
I20260928 06:42:49.842339 2105 webserver.cc:460] Webserver started at http://127.18.211.126:40407/ using document root <none> and password file <none>
I20260928 06:42:49.842548 2105 fs_manager.cc:359] Metadata directory not provided
I20260928 06:42:49.842594 2105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:42:49.842717 2105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:42:49.843775 2105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "f0cbe448c598422385926ffb3fef2e98"
format_stamp: "Formatted at 2026-09-28 06:42:49 on dist-test-slave-b28r"
server_key: "4b5b1a3974a136b611475cfff92cc3ae"
server_key_iv: "5ad50cc53ae0c0032e5abbdae82f41c6"
server_key_version: "encryptionkey@0"
I20260928 06:42:49.844107 2105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "f0cbe448c598422385926ffb3fef2e98"
format_stamp: "Formatted at 2026-09-28 06:42:49 on dist-test-slave-b28r"
server_key: "4b5b1a3974a136b611475cfff92cc3ae"
server_key_iv: "5ad50cc53ae0c0032e5abbdae82f41c6"
server_key_version: "encryptionkey@0"
I20260928 06:42:49.845436 2105 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:42:49.846148 2120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:49.846313 2105 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:42:49.846393 2105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "f0cbe448c598422385926ffb3fef2e98"
format_stamp: "Formatted at 2026-09-28 06:42:49 on dist-test-slave-b28r"
server_key: "4b5b1a3974a136b611475cfff92cc3ae"
server_key_iv: "5ad50cc53ae0c0032e5abbdae82f41c6"
server_key_version: "encryptionkey@0"
I20260928 06:42:49.846454 2105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:42:49.862386 2105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:42:49.864133 2105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:42:49.864261 2105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:42:49.867947 2105 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:35839
I20260928 06:42:49.867980 2173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:35839 every 8 connection(s)
I20260928 06:42:49.868283 2105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:42:49.869059 2174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:42:49.871275 2174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98: Bootstrap starting.
I20260928 06:42:49.871829 2174 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98: Neither blocks nor log segments found. Creating new log.
I20260928 06:42:49.872090 2174 log.cc:824] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98: Log is configured to *not* fsync() on all Append() calls
I20260928 06:42:49.872694 2174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98: No bootstrap required, opened a new log
I20260928 06:42:49.874228 2174 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0cbe448c598422385926ffb3fef2e98" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35839 } }
I20260928 06:42:49.874362 2174 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:42:49.874383 2174 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0cbe448c598422385926ffb3fef2e98, State: Initialized, Role: FOLLOWER
I20260928 06:42:49.874521 2174 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0cbe448c598422385926ffb3fef2e98" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35839 } }
I20260928 06:42:49.874583 2174 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:42:49.874604 2174 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:42:49.874650 2174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:42:49.875155 2174 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0cbe448c598422385926ffb3fef2e98" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35839 } }
I20260928 06:42:49.875245 2174 leader_election.cc:302] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0cbe448c598422385926ffb3fef2e98; no voters:
I20260928 06:42:49.875396 2174 leader_election.cc:288] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:42:49.875437 2177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:42:49.875578 2177 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [term 1 LEADER]: Becoming Leader. State: Replica: f0cbe448c598422385926ffb3fef2e98, State: Running, Role: LEADER
I20260928 06:42:49.875665 2174 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:42:49.875695 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 2105
I20260928 06:42:49.875790 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:42:49.875712 2177 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0cbe448c598422385926ffb3fef2e98" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35839 } }
I20260928 06:42:49.875993 19277 external_mini_cluster.cc:1428] Setting key 617130135e8b1c9c3b6d76d5d306e984
I20260928 06:42:49.876222 2177 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0cbe448c598422385926ffb3fef2e98. Latest consensus state: current_term: 1 leader_uuid: "f0cbe448c598422385926ffb3fef2e98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0cbe448c598422385926ffb3fef2e98" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35839 } } }
I20260928 06:42:49.876228 2179 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0cbe448c598422385926ffb3fef2e98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0cbe448c598422385926ffb3fef2e98" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35839 } } }
I20260928 06:42:49.876319 2177 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [sys.catalog]: This master's current role is: LEADER
I20260928 06:42:49.876324 2179 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [sys.catalog]: This master's current role is: LEADER
I20260928 06:42:49.876524 2186 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:42:49.877034 2186 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 06:42:49 dist-test-slave-b28r krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:42:49.884246 2186 catalog_manager.cc:1347] Generated new cluster ID: a098db18aaa54e6d84d051b51772f69a
I20260928 06:42:49.884325 2186 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:42:49.886572 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:49.876643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53916 (local address 127.18.211.126:35839)
0928 06:42:49.876776 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:49.876786 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:49.876846 (+ 60us) server_negotiation.cc:407] Connection header received
0928 06:42:49.876962 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:49.876969 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:49.877025 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:49.878009 (+ 984us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:49.878155 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:49.879038 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:49.880146 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:49.880571 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:49.880648 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:49.883356 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:49.883362 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:49.883362 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:49.883378 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:49.885095 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:49.885443 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:49.885446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:49.885446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:49.885538 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:49.885712 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:49.885713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:49.885713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:49.885965 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:49.886097 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:49.886211 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 06:42:49.886360 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":60,"threads_started":1}
I20260928 06:42:49.897754 2186 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:42:49.898475 2186 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:42:49.902987 2186 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98: Generated new TSK 0
I20260928 06:42:49.903187 2186 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.211.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab.
Entry for principal kudu/127.18.211.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab.
I20260928 06:42:49.929528 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.65:0
--local_ip_for_outbound_sockets=127.18.211.65
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35839
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:42:50.009411 2200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:42:50.009624 2200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:42:50.009655 2200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:42:50.010807 2200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:42:50.010847 2200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:42:50.010881 2200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.65
I20260928 06:42:50.012343 2200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.65
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:42:50.012573 2200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:42:50.012789 2200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:42:50.017596 2208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:50.017588 2206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:50.017589 2205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:42:50.017827 2200 server_base.cc:1034] running on GCE node
I20260928 06:42:50.018055 2200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:42:50.018285 2200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:42:50.019439 2200 hybrid_clock.cc:648] HybridClock initialized: now 1790577770019424 us; error 30 us; skew 500 ppm
Sep 28 06:42:50 dist-test-slave-b28r krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577770, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:42:50.021862 2200 init.cc:377] Logged in from keytab as kudu/127.18.211.65@KRBTEST.COM (short username kudu)
I20260928 06:42:50.022244 2200 webserver.cc:460] Webserver started at http://127.18.211.65:33261/ using document root <none> and password file <none>
I20260928 06:42:50.022430 2200 fs_manager.cc:359] Metadata directory not provided
I20260928 06:42:50.022471 2200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:42:50.022588 2200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:42:50.023605 2200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/data/instance:
uuid: "5dd26eed1e5243c7b65f80d825b02db3"
format_stamp: "Formatted at 2026-09-28 06:42:50 on dist-test-slave-b28r"
server_key: "71f571bee70f0fc6d587479b21c5e2fb"
server_key_iv: "c8e57d9b01fc0c7b30b19354ed03964f"
server_key_version: "encryptionkey@0"
I20260928 06:42:50.023926 2200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance:
uuid: "5dd26eed1e5243c7b65f80d825b02db3"
format_stamp: "Formatted at 2026-09-28 06:42:50 on dist-test-slave-b28r"
server_key: "71f571bee70f0fc6d587479b21c5e2fb"
server_key_iv: "c8e57d9b01fc0c7b30b19354ed03964f"
server_key_version: "encryptionkey@0"
I20260928 06:42:50.025147 2200 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:42:50.025866 2215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:50.026028 2200 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:42:50.026108 2200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/wal
uuid: "5dd26eed1e5243c7b65f80d825b02db3"
format_stamp: "Formatted at 2026-09-28 06:42:50 on dist-test-slave-b28r"
server_key: "71f571bee70f0fc6d587479b21c5e2fb"
server_key_iv: "c8e57d9b01fc0c7b30b19354ed03964f"
server_key_version: "encryptionkey@0"
I20260928 06:42:50.026166 2200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:42:50.071774 2200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:42:50.074023 2200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:42:50.074162 2200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:42:50.074406 2200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:42:50.074733 2200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:42:50.074770 2200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:50.074800 2200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:42:50.074816 2200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:50.080518 2200 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.65:40593
I20260928 06:42:50.080582 2328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.65:40593 every 8 connection(s)
I20260928 06:42:50.080876 2200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
I20260928 06:42:50.085004 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 2200
I20260928 06:42:50.085106 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance
I20260928 06:42:50.085347 19277 external_mini_cluster.cc:1428] Setting key 5bdf5b94cd2525ecffad6db10befc8d1
Sep 28 06:42:50 dist-test-slave-b28r krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577770, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:42:50.090135 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:50.081532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:60861 (local address 127.18.211.126:35839)
0928 06:42:50.081586 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:50.081588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:50.081798 (+ 210us) server_negotiation.cc:407] Connection header received
0928 06:42:50.082259 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:50.082260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:50.082275 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:50.082327 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:50.083276 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:50.083791 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:50.084806 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:50.085083 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:50.085153 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:50.087388 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:50.087389 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:50.087390 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:50.087397 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:50.088824 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:50.089185 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:50.089186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:50.089187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:50.089248 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:50.089455 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:50.089455 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:50.089456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:50.089584 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:50.089651 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:50.089941 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 06:42:50.090037 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 06:42:50.090184 2331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:50.081509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.65:60861)
0928 06:42:50.081665 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:50.081704 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:50.082175 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:50.082387 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:50.082390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:50.082934 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:42:50.083155 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:50.083161 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:50.083900 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:50.083901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:50.084718 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:50.084721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:50.085165 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:50.085166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:50.085334 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:50.085796 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:42:50.085806 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:42:50.087302 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:42:50.088898 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:50.088905 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:50.088907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:50.089118 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:50.089304 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:50.089305 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:50.089305 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:50.089402 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:50.089639 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:42:50.089643 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:42:50.089812 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 06:42:50.089899 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 06:42:50.090024 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260928 06:42:50.090653 2329 heartbeater.cc:347] Connected to a master server at 127.18.211.126:35839
I20260928 06:42:50.090747 2329 heartbeater.cc:464] Registering TS with master...
I20260928 06:42:50.090925 2329 heartbeater.cc:511] Master 127.18.211.126:35839 requested a full tablet report, sending...
I20260928 06:42:50.091440 2137 ts_manager.cc:194] Registered new tserver with Master: 5dd26eed1e5243c7b65f80d825b02db3 (127.18.211.65:40593)
I20260928 06:42:50.092003 2137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.65@KRBTEST.COM'} at 127.18.211.65:60861
WARNING: no policy specified for kudu/127.18.211.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab.
Entry for principal kudu/127.18.211.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab.
I20260928 06:42:50.108083 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.66:0
--local_ip_for_outbound_sockets=127.18.211.66
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35839
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:42:50.187269 2334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:42:50.187471 2334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:42:50.187502 2334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:42:50.188711 2334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:42:50.188764 2334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:42:50.188802 2334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.66
I20260928 06:42:50.190220 2334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.66
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:42:50.190451 2334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:42:50.190657 2334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:42:50.195397 2340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:50.195387 2342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:42:50.195402 2334 server_base.cc:1034] running on GCE node
W20260928 06:42:50.195387 2339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:42:50.195801 2334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:42:50.196027 2334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:42:50.197177 2334 hybrid_clock.cc:648] HybridClock initialized: now 1790577770197160 us; error 30 us; skew 500 ppm
Sep 28 06:42:50 dist-test-slave-b28r krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577770, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:42:50.199620 2334 init.cc:377] Logged in from keytab as kudu/127.18.211.66@KRBTEST.COM (short username kudu)
I20260928 06:42:50.200065 2334 webserver.cc:460] Webserver started at http://127.18.211.66:33867/ using document root <none> and password file <none>
I20260928 06:42:50.200259 2334 fs_manager.cc:359] Metadata directory not provided
I20260928 06:42:50.200307 2334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:42:50.200429 2334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:42:50.201444 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/data/instance:
uuid: "1b9d2814da9c4fe89992a72a0bbb858c"
format_stamp: "Formatted at 2026-09-28 06:42:50 on dist-test-slave-b28r"
server_key: "62ed72ee70ef531d283b0d048a7d09bd"
server_key_iv: "86648977cc16e351de47fc7474884780"
server_key_version: "encryptionkey@0"
I20260928 06:42:50.201771 2334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance:
uuid: "1b9d2814da9c4fe89992a72a0bbb858c"
format_stamp: "Formatted at 2026-09-28 06:42:50 on dist-test-slave-b28r"
server_key: "62ed72ee70ef531d283b0d048a7d09bd"
server_key_iv: "86648977cc16e351de47fc7474884780"
server_key_version: "encryptionkey@0"
I20260928 06:42:50.203047 2334 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:42:50.203783 2349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:50.203948 2334 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:42:50.204028 2334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/wal
uuid: "1b9d2814da9c4fe89992a72a0bbb858c"
format_stamp: "Formatted at 2026-09-28 06:42:50 on dist-test-slave-b28r"
server_key: "62ed72ee70ef531d283b0d048a7d09bd"
server_key_iv: "86648977cc16e351de47fc7474884780"
server_key_version: "encryptionkey@0"
I20260928 06:42:50.204093 2334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:42:50.226861 2334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:42:50.228783 2334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:42:50.228897 2334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:42:50.229066 2334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:42:50.229382 2334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:42:50.229417 2334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:50.229439 2334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:42:50.229462 2334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:50.235006 2334 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.66:33735
I20260928 06:42:50.235072 2464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.66:33735 every 8 connection(s)
I20260928 06:42:50.235375 2334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
Sep 28 06:42:50 dist-test-slave-b28r krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577770, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:42:50.242687 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 2334
I20260928 06:42:50.242785 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance
I20260928 06:42:50.242971 19277 external_mini_cluster.cc:1428] Setting key 48c758c45ac579370211272ea0572397
I20260928 06:42:50.244378 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:50.236048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:57235 (local address 127.18.211.126:35839)
0928 06:42:50.236104 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:50.236106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:50.236297 (+ 191us) server_negotiation.cc:407] Connection header received
0928 06:42:50.236841 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:50.236843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:50.236853 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:50.236918 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:50.237798 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:50.238306 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:50.239302 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:50.239498 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:50.239551 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:50.241457 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:50.241459 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:50.241459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:50.241465 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:50.242944 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:50.243383 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:50.243384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:50.243385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:50.243450 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:50.243768 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:50.243769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:50.243769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:50.243878 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:50.243926 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:50.244234 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 06:42:50.244313 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 06:42:50.244617 2467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:50.236016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.66:57235)
0928 06:42:50.236172 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:50.236212 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:50.236686 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:50.236972 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:50.236975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:50.237517 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:42:50.237704 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:50.237709 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:50.238379 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:50.238380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:50.239226 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:50.239230 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:50.239558 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:50.239558 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:50.239651 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:50.240044 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:42:50.240051 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:42:50.241375 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:42:50.243030 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:50.243040 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:50.243042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:50.243323 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:50.243540 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:50.243541 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:50.243541 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:50.243657 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:50.243939 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:42:50.243946 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:42:50.244103 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 06:42:50.244221 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 06:42:50.244394 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260928 06:42:50.245108 2465 heartbeater.cc:347] Connected to a master server at 127.18.211.126:35839
I20260928 06:42:50.245221 2465 heartbeater.cc:464] Registering TS with master...
I20260928 06:42:50.245428 2465 heartbeater.cc:511] Master 127.18.211.126:35839 requested a full tablet report, sending...
I20260928 06:42:50.245831 2137 ts_manager.cc:194] Registered new tserver with Master: 1b9d2814da9c4fe89992a72a0bbb858c (127.18.211.66:33735)
I20260928 06:42:50.246313 2137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.66@KRBTEST.COM'} at 127.18.211.66:57235
WARNING: no policy specified for kudu/127.18.211.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab.
Entry for principal kudu/127.18.211.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab.
I20260928 06:42:50.265576 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.67:0
--local_ip_for_outbound_sockets=127.18.211.67
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35839
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:42:50.344290 2470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:42:50.344485 2470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:42:50.344507 2470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:42:50.345582 2470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:42:50.345614 2470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:42:50.345638 2470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.67
I20260928 06:42:50.346983 2470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2470
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.67
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:42:50.347220 2470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:42:50.347426 2470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:42:50.352212 2478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:50.352233 2476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:42:50.352234 2475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:42:50.352355 2470 server_base.cc:1034] running on GCE node
I20260928 06:42:50.352644 2470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:42:50.352907 2470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:42:50.354050 2470 hybrid_clock.cc:648] HybridClock initialized: now 1790577770354034 us; error 33 us; skew 500 ppm
Sep 28 06:42:50 dist-test-slave-b28r krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577770, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:42:50.356559 2470 init.cc:377] Logged in from keytab as kudu/127.18.211.67@KRBTEST.COM (short username kudu)
I20260928 06:42:50.356985 2470 webserver.cc:460] Webserver started at http://127.18.211.67:40923/ using document root <none> and password file <none>
I20260928 06:42:50.357177 2470 fs_manager.cc:359] Metadata directory not provided
I20260928 06:42:50.357231 2470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:42:50.357349 2470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:42:50.358363 2470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/data/instance:
uuid: "b745a811fae54cd996e42b4fc6f14864"
format_stamp: "Formatted at 2026-09-28 06:42:50 on dist-test-slave-b28r"
server_key: "a11ae81d480488b6272d098a3c4f6043"
server_key_iv: "e99f696f5a2faebcacebde03a8571912"
server_key_version: "encryptionkey@0"
I20260928 06:42:50.358695 2470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance:
uuid: "b745a811fae54cd996e42b4fc6f14864"
format_stamp: "Formatted at 2026-09-28 06:42:50 on dist-test-slave-b28r"
server_key: "a11ae81d480488b6272d098a3c4f6043"
server_key_iv: "e99f696f5a2faebcacebde03a8571912"
server_key_version: "encryptionkey@0"
I20260928 06:42:50.360038 2470 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:42:50.360812 2485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:50.360986 2470 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:42:50.361068 2470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/wal
uuid: "b745a811fae54cd996e42b4fc6f14864"
format_stamp: "Formatted at 2026-09-28 06:42:50 on dist-test-slave-b28r"
server_key: "a11ae81d480488b6272d098a3c4f6043"
server_key_iv: "e99f696f5a2faebcacebde03a8571912"
server_key_version: "encryptionkey@0"
I20260928 06:42:50.361129 2470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:42:50.381770 2470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:42:50.383546 2470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:42:50.383677 2470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:42:50.383886 2470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:42:50.384181 2470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:42:50.384214 2470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:50.384243 2470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:42:50.384266 2470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:42:50.389915 2470 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.67:40893
I20260928 06:42:50.389972 2600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.67:40893 every 8 connection(s)
I20260928 06:42:50.390266 2470 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:42:50 dist-test-slave-b28r krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577770, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:42:50.398962 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:50.390949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:55497 (local address 127.18.211.126:35839)
0928 06:42:50.391003 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:50.391005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:50.391220 (+ 215us) server_negotiation.cc:407] Connection header received
0928 06:42:50.391723 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:50.391725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:50.391736 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:50.391813 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:50.392711 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:50.393217 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:50.394208 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:50.394412 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:50.394468 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:50.396523 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:50.396526 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:50.396526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:50.396533 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:50.397656 (+ 1123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:50.398079 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:50.398080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:50.398081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:50.398135 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:50.398354 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:50.398355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:50.398355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:50.398465 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:50.398519 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:50.398786 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 06:42:50.398861 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:42:50.399142 2603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:50.390920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.67:55497)
0928 06:42:50.391075 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:50.391140 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:50.391612 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:50.391878 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:50.391882 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:50.392427 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:42:50.392623 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:50.392628 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:50.393304 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:50.393304 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:50.394130 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:50.394133 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:50.394475 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:50.394476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:50.394571 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:50.395001 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:42:50.395010 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:42:50.396421 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:42:50.397744 (+ 1323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:50.397751 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:50.397753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:50.398012 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:50.398189 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:42:50.398190 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:42:50.398191 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:42:50.398300 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:42:50.398519 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:42:50.398526 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:42:50.398655 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 06:42:50.398756 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 06:42:50.398907 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20260928 06:42:50.399592 2601 heartbeater.cc:347] Connected to a master server at 127.18.211.126:35839
I20260928 06:42:50.399690 2601 heartbeater.cc:464] Registering TS with master...
I20260928 06:42:50.399855 2601 heartbeater.cc:511] Master 127.18.211.126:35839 requested a full tablet report, sending...
I20260928 06:42:50.400022 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 2470
I20260928 06:42:50.400106 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance
I20260928 06:42:50.400280 19277 external_mini_cluster.cc:1428] Setting key 8b30c237622ea29c0d0723a016654a69
I20260928 06:42:50.400265 2137 ts_manager.cc:194] Registered new tserver with Master: b745a811fae54cd996e42b4fc6f14864 (127.18.211.67:40893)
I20260928 06:42:50.400714 2137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.67@KRBTEST.COM'} at 127.18.211.67:55497
I20260928 06:42:50.401068 19277 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:42:50.407478 2189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:50.402219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53920 (local address 127.18.211.126:35839)
0928 06:42:50.402264 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:50.402266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:50.402397 (+ 131us) server_negotiation.cc:407] Connection header received
0928 06:42:50.402434 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:50.402435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:50.402443 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:50.402476 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:42:50.402978 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:50.403519 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:50.404261 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:50.404442 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:50.404490 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:50.405463 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:42:50.405464 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:42:50.405465 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:42:50.405470 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:42:50.406463 (+ 993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:50.406779 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:50.406780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:50.406780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:50.406828 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:42:50.406971 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:42:50.406971 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:42:50.406972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:42:50.407124 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:42:50.407197 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:50.407366 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 06:42:50.407421 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:42:50.409142 2136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:42:50.410705 2136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:42:50.416814 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:50.412851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:40893 (local address 127.0.0.1:53798)
0928 06:42:50.413131 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:50.413173 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:50.413243 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:50.413445 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:50.413448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:50.413450 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:42:50.413507 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:50.413509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:50.414526 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:50.414527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:50.415643 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:50.415646 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:50.416564 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:50.416564 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:50.416628 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:50.416636 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 06:42:50.416690 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 06:42:50.416742 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":60,"threads_started":1}
I20260928 06:42:50.416846 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:50.412945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53798 (local address 127.18.211.67:40893)
0928 06:42:50.413132 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:50.413137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:50.413259 (+ 122us) server_negotiation.cc:407] Connection header received
0928 06:42:50.413332 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:50.413335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:50.413351 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:50.413437 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:42:50.413574 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:50.414422 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:50.415804 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:50.416447 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:50.416481 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:50.416518 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:50.416723 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 06:42:50.416774 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":95,"threads_started":1}
I20260928 06:42:50.417172 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:50.413051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:33735 (local address 127.0.0.1:35038)
0928 06:42:50.413354 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:50.413362 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:50.413395 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:50.413802 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:50.413803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:50.413805 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:42:50.413834 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:50.413836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:50.415110 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:50.415111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:50.416014 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:50.416017 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:50.416946 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:50.416947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:50.417005 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:50.417009 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:42:50.417052 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 06:42:50.417090 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":31,"thread_start_us":78,"threads_started":1}
I20260928 06:42:50.417375 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:50.413205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35038 (local address 127.18.211.66:33735)
0928 06:42:50.413568 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:50.413572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:50.413582 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:42:50.413617 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:50.413619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:50.413632 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:50.413719 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:42:50.413907 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:50.414987 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:50.416085 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:50.416788 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:50.416857 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:50.416908 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:50.417252 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 06:42:50.417298 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":91,"threads_started":1}
I20260928 06:42:50.417251 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:50.412851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:40593 (local address 127.0.0.1:34190)
0928 06:42:50.413179 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:50.413189 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:50.413243 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:50.413676 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:50.413677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:50.413680 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:42:50.413732 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:50.413734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:50.415044 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:50.415045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:50.416154 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:50.416157 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:50.416996 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:50.416997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:50.417055 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:50.417059 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:42:50.417112 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 06:42:50.417150 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":3,"thread_start_us":69,"threads_started":1}
I20260928 06:42:50.417881 2534 tablet_service.cc:1467] Processing CreateTablet for tablet e6f202f8f2ad456386d9bb5d5e3070c5 (DEFAULT_TABLE table=test-table [id=1e82901d5b8f46a1aa88d670eb22d624]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:42:50.417940 2397 tablet_service.cc:1467] Processing CreateTablet for tablet e6f202f8f2ad456386d9bb5d5e3070c5 (DEFAULT_TABLE table=test-table [id=1e82901d5b8f46a1aa88d670eb22d624]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:42:50.418246 2534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6f202f8f2ad456386d9bb5d5e3070c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:42:50.418290 2397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6f202f8f2ad456386d9bb5d5e3070c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:42:50.417236 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:50.412949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34190 (local address 127.18.211.65:40593)
0928 06:42:50.413443 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:50.413449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:50.413459 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:42:50.413502 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:50.413504 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:50.413520 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:50.413591 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:42:50.413806 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:50.414932 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:50.416235 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:50.416931 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:50.416994 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:50.417042 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:50.417124 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 06:42:50.417163 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":90,"threads_started":1}
I20260928 06:42:50.419724 2263 tablet_service.cc:1467] Processing CreateTablet for tablet e6f202f8f2ad456386d9bb5d5e3070c5 (DEFAULT_TABLE table=test-table [id=1e82901d5b8f46a1aa88d670eb22d624]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:42:50.420048 2263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6f202f8f2ad456386d9bb5d5e3070c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:42:50.420854 2618 tablet_bootstrap.cc:492] T e6f202f8f2ad456386d9bb5d5e3070c5 P b745a811fae54cd996e42b4fc6f14864: Bootstrap starting.
I20260928 06:42:50.421276 2619 tablet_bootstrap.cc:492] T e6f202f8f2ad456386d9bb5d5e3070c5 P 1b9d2814da9c4fe89992a72a0bbb858c: Bootstrap starting.
I20260928 06:42:50.421717 2618 tablet_bootstrap.cc:654] T e6f202f8f2ad456386d9bb5d5e3070c5 P b745a811fae54cd996e42b4fc6f14864: Neither blocks nor log segments found. Creating new log.
I20260928 06:42:50.421892 2619 tablet_bootstrap.cc:654] T e6f202f8f2ad456386d9bb5d5e3070c5 P 1b9d2814da9c4fe89992a72a0bbb858c: Neither blocks nor log segments found. Creating new log.
I20260928 06:42:50.422309 2618 log.cc:824] T e6f202f8f2ad456386d9bb5d5e3070c5 P b745a811fae54cd996e42b4fc6f14864: Log is configured to *not* fsync() on all Append() calls
I20260928 06:42:50.422387 2619 log.cc:824] T e6f202f8f2ad456386d9bb5d5e3070c5 P 1b9d2814da9c4fe89992a72a0bbb858c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:42:50.423056 2619 tablet_bootstrap.cc:492] T e6f202f8f2ad456386d9bb5d5e3070c5 P 1b9d2814da9c4fe89992a72a0bbb858c: No bootstrap required, opened a new log
I20260928 06:42:50.423069 2618 tablet_bootstrap.cc:492] T e6f202f8f2ad456386d9bb5d5e3070c5 P b745a811fae54cd996e42b4fc6f14864: No bootstrap required, opened a new log
I20260928 06:42:50.423139 2619 ts_tablet_manager.cc:1397] T e6f202f8f2ad456386d9bb5d5e3070c5 P 1b9d2814da9c4fe89992a72a0bbb858c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:42:50.423158 2618 ts_tablet_manager.cc:1397] T e6f202f8f2ad456386d9bb5d5e3070c5 P b745a811fae54cd996e42b4fc6f14864: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:42:50.424412 2619 raft_consensus.cc:348] T e6f202f8f2ad456386d9bb5d5e3070c5 P 1b9d2814da9c4fe89992a72a0bbb858c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b745a811fae54cd996e42b4fc6f14864" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40893 } } peers { permanent_uuid: "5dd26eed1e5243c7b65f80d825b02db3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 40593 } } peers { permanent_uuid: "1b9d2814da9c4fe89992a72a0bbb858c" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33735 } }
I20260928 06:42:50.424542 2619 raft_consensus.cc:374] T e6f202f8f2ad456386d9bb5d5e3070c5 P 1b9d2814da9c4fe89992a72a0bbb858c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:42:50.424451 2618 raft_consensus.cc:348] T e6f202f8f2ad456386d9bb5d5e3070c5 P b745a811fae54cd996e42b4fc6f14864 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b745a811fae54cd996e42b4fc6f14864" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40893 } } peers { permanent_uuid: "5dd26eed1e5243c7b65f80d825b02db3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 40593 } } peers { permanent_uuid: "1b9d2814da9c4fe89992a72a0bbb858c" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33735 } }
I20260928 06:42:50.424563 2619 raft_consensus.cc:729] T e6f202f8f2ad456386d9bb5d5e3070c5 P 1b9d2814da9c4fe89992a72a0bbb858c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b9d2814da9c4fe89992a72a0bbb858c, State: Initialized, Role: FOLLOWER
I20260928 06:42:50.424575 2618 raft_consensus.cc:374] T e6f202f8f2ad456386d9bb5d5e3070c5 P b745a811fae54cd996e42b4fc6f14864 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:42:50.424629 2618 raft_consensus.cc:729] T e6f202f8f2ad456386d9bb5d5e3070c5 P b745a811fae54cd996e42b4fc6f14864 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b745a811fae54cd996e42b4fc6f14864, State: Initialized, Role: FOLLOWER
I20260928 06:42:50.424685 2619 consensus_queue.cc:260] T e6f202f8f2ad456386d9bb5d5e3070c5 P 1b9d2814da9c4fe89992a72a0bbb858c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b745a811fae54cd996e42b4fc6f14864" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40893 } } peers { permanent_uuid: "5dd26eed1e5243c7b65f80d825b02db3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 40593 } } peers { permanent_uuid: "1b9d2814da9c4fe89992a72a0bbb858c" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33735 } }
I20260928 06:42:50.424737 2618 consensus_queue.cc:260] T e6f202f8f2ad456386d9bb5d5e3070c5 P b745a811fae54cd996e42b4fc6f14864 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b745a811fae54cd996e42b4fc6f14864" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40893 } } peers { permanent_uuid: "5dd26eed1e5243c7b65f80d825b02db3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 40593 } } peers { permanent_uuid: "1b9d2814da9c4fe89992a72a0bbb858c" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33735 } }
I20260928 06:42:50.424947 2619 ts_tablet_manager.cc:1428] T e6f202f8f2ad456386d9bb5d5e3070c5 P 1b9d2814da9c4fe89992a72a0bbb858c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:42:50.424983 2465 heartbeater.cc:503] Master 127.18.211.126:35839 was elected leader, sending a full tablet report...
I20260928 06:42:50.425038 2601 heartbeater.cc:503] Master 127.18.211.126:35839 was elected leader, sending a full tablet report...
I20260928 06:42:50.425123 2622 tablet_bootstrap.cc:492] T e6f202f8f2ad456386d9bb5d5e3070c5 P 5dd26eed1e5243c7b65f80d825b02db3: Bootstrap starting.
I20260928 06:42:50.425027 2618 ts_tablet_manager.cc:1428] T e6f202f8f2ad456386d9bb5d5e3070c5 P b745a811fae54cd996e42b4fc6f14864: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:42:50.425686 2622 tablet_bootstrap.cc:654] T e6f202f8f2ad456386d9bb5d5e3070c5 P 5dd26eed1e5243c7b65f80d825b02db3: Neither blocks nor log segments found. Creating new log.
I20260928 06:42:50.426250 2622 log.cc:824] T e6f202f8f2ad456386d9bb5d5e3070c5 P 5dd26eed1e5243c7b65f80d825b02db3: Log is configured to *not* fsync() on all Append() calls
I20260928 06:42:50.426855 2622 tablet_bootstrap.cc:492] T e6f202f8f2ad456386d9bb5d5e3070c5 P 5dd26eed1e5243c7b65f80d825b02db3: No bootstrap required, opened a new log
I20260928 06:42:50.426914 2622 ts_tablet_manager.cc:1397] T e6f202f8f2ad456386d9bb5d5e3070c5 P 5dd26eed1e5243c7b65f80d825b02db3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:42:50.428392 2622 raft_consensus.cc:348] T e6f202f8f2ad456386d9bb5d5e3070c5 P 5dd26eed1e5243c7b65f80d825b02db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b745a811fae54cd996e42b4fc6f14864" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40893 } } peers { permanent_uuid: "5dd26eed1e5243c7b65f80d825b02db3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 40593 } } peers { permanent_uuid: "1b9d2814da9c4fe89992a72a0bbb858c" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33735 } }
I20260928 06:42:50.428514 2622 raft_consensus.cc:374] T e6f202f8f2ad456386d9bb5d5e3070c5 P 5dd26eed1e5243c7b65f80d825b02db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:42:50.428534 2622 raft_consensus.cc:729] T e6f202f8f2ad456386d9bb5d5e3070c5 P 5dd26eed1e5243c7b65f80d825b02db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd26eed1e5243c7b65f80d825b02db3, State: Initialized, Role: FOLLOWER
I20260928 06:42:50.428633 2622 consensus_queue.cc:260] T e6f202f8f2ad456386d9bb5d5e3070c5 P 5dd26eed1e5243c7b65f80d825b02db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b745a811fae54cd996e42b4fc6f14864" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40893 } } peers { permanent_uuid: "5dd26eed1e5243c7b65f80d825b02db3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 40593 } } peers { permanent_uuid: "1b9d2814da9c4fe89992a72a0bbb858c" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33735 } }
I20260928 06:42:50.428865 2622 ts_tablet_manager.cc:1428] T e6f202f8f2ad456386d9bb5d5e3070c5 P 5dd26eed1e5243c7b65f80d825b02db3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:42:50.428921 2329 heartbeater.cc:503] Master 127.18.211.126:35839 was elected leader, sending a full tablet report...
W20260928 06:42:50.485934 2466 tablet.cc:2369] T e6f202f8f2ad456386d9bb5d5e3070c5 P 1b9d2814da9c4fe89992a72a0bbb858c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:42:50.520793 2624 raft_consensus.cc:482] T e6f202f8f2ad456386d9bb5d5e3070c5 P b745a811fae54cd996e42b4fc6f14864 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:42:50.520936 2624 raft_consensus.cc:504] T e6f202f8f2ad456386d9bb5d5e3070c5 P b745a811fae54cd996e42b4fc6f14864 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b745a811fae54cd996e42b4fc6f14864" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40893 } } peers { permanent_uuid: "5dd26eed1e5243c7b65f80d825b02db3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 40593 } } peers { permanent_uuid: "1b9d2814da9c4fe89992a72a0bbb858c" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33735 } }
I20260928 06:42:50.521320 2624 leader_election.cc:288] T e6f202f8f2ad456386d9bb5d5e3070c5 P b745a811fae54cd996e42b4fc6f14864 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dd26eed1e5243c7b65f80d825b02db3 (127.18.211.65:40593), 1b9d2814da9c4fe89992a72a0bbb858c (127.18.211.66:33735)
I20260928 06:42:50.525269 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:50.521650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:44215 (local address 127.18.211.65:40593)
0928 06:42:50.521719 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:50.521721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:50.521822 (+ 101us) server_negotiation.cc:407] Connection header received
0928 06:42:50.521857 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:50.521858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:50.521869 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:50.521912 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:42:50.522046 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:50.522763 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:50.524289 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:50.524891 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:50.524957 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:50.524966 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:50.525150 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 06:42:50.525196 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:42:50.525265 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:50.521527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:40593 (local address 127.18.211.67:44215)
0928 06:42:50.521744 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:50.521759 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:50.521816 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:50.521930 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:50.521931 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:50.521932 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:42:50.521992 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:50.521994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:50.522859 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:50.522860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:50.524192 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:50.524196 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:50.524964 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:50.524965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:50.525052 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:50.525056 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:42:50.525113 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 06:42:50.525161 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":53,"thread_start_us":111,"threads_started":1}
I20260928 06:42:50.525302 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:50.521617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:55075 (local address 127.18.211.66:33735)
0928 06:42:50.521665 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:50.521668 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:50.521697 (+ 29us) server_negotiation.cc:407] Connection header received
0928 06:42:50.521727 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:50.521728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:50.521738 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:50.521783 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:42:50.521960 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:50.522747 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:50.524304 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:50.524883 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:50.524961 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:50.524968 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:50.525150 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 06:42:50.525196 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:42:50.525321 2603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:50.521512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:33735 (local address 127.18.211.67:55075)
0928 06:42:50.521570 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:50.521585 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 06:42:50.521649 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:42:50.521814 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:42:50.521815 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:42:50.521817 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:42:50.521891 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:50.521893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:50.522859 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:50.522860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:50.524192 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:42:50.524196 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:42:50.524964 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:42:50.524965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:42:50.525055 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:50.525057 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:42:50.525113 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 06:42:50.525162 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":96}
I20260928 06:42:50.525646 2283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6f202f8f2ad456386d9bb5d5e3070c5" candidate_uuid: "b745a811fae54cd996e42b4fc6f14864" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd26eed1e5243c7b65f80d825b02db3" is_pre_election: true
I20260928 06:42:50.525671 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6f202f8f2ad456386d9bb5d5e3070c5" candidate_uuid: "b745a811fae54cd996e42b4fc6f14864" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b9d2814da9c4fe89992a72a0bbb858c" is_pre_election: true
I20260928 06:42:50.525817 2417 raft_consensus.cc:2454] T e6f202f8f2ad456386d9bb5d5e3070c5 P 1b9d2814da9c4fe89992a72a0bbb858c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b745a811fae54cd996e42b4fc6f14864 in term 0.
I20260928 06:42:50.525822 2283 raft_consensus.cc:2454] T e6f202f8f2ad456386d9bb5d5e3070c5 P 5dd26eed1e5243c7b65f80d825b02db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b745a811fae54cd996e42b4fc6f14864 in term 0.
I20260928 06:42:50.526048 2489 leader_election.cc:302] T e6f202f8f2ad456386d9bb5d5e3070c5 P b745a811fae54cd996e42b4fc6f14864 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b9d2814da9c4fe89992a72a0bbb858c, b745a811fae54cd996e42b4fc6f14864; no voters:
I20260928 06:42:50.526211 2624 raft_consensus.cc:2781] T e6f202f8f2ad456386d9bb5d5e3070c5 P b745a811fae54cd996e42b4fc6f14864 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:42:50.526273 2624 raft_consensus.cc:482] T e6f202f8f2ad456386d9bb5d5e3070c5 P b745a811fae54cd996e42b4fc6f14864 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:42:50.526289 2624 raft_consensus.cc:3037] T e6f202f8f2ad456386d9bb5d5e3070c5 P b745a811fae54cd996e42b4fc6f14864 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:42:50.526994 2624 raft_consensus.cc:504] T e6f202f8f2ad456386d9bb5d5e3070c5 P b745a811fae54cd996e42b4fc6f14864 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b745a811fae54cd996e42b4fc6f14864" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40893 } } peers { permanent_uuid: "5dd26eed1e5243c7b65f80d825b02db3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 40593 } } peers { permanent_uuid: "1b9d2814da9c4fe89992a72a0bbb858c" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33735 } }
I20260928 06:42:50.527143 2624 leader_election.cc:288] T e6f202f8f2ad456386d9bb5d5e3070c5 P b745a811fae54cd996e42b4fc6f14864 [CANDIDATE]: Term 1 election: Requested vote from peers 5dd26eed1e5243c7b65f80d825b02db3 (127.18.211.65:40593), 1b9d2814da9c4fe89992a72a0bbb858c (127.18.211.66:33735)
I20260928 06:42:50.527334 2417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6f202f8f2ad456386d9bb5d5e3070c5" candidate_uuid: "b745a811fae54cd996e42b4fc6f14864" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b9d2814da9c4fe89992a72a0bbb858c"
I20260928 06:42:50.527288 2283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6f202f8f2ad456386d9bb5d5e3070c5" candidate_uuid: "b745a811fae54cd996e42b4fc6f14864" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dd26eed1e5243c7b65f80d825b02db3"
I20260928 06:42:50.527415 2417 raft_consensus.cc:3037] T e6f202f8f2ad456386d9bb5d5e3070c5 P 1b9d2814da9c4fe89992a72a0bbb858c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:42:50.527490 2283 raft_consensus.cc:3037] T e6f202f8f2ad456386d9bb5d5e3070c5 P 5dd26eed1e5243c7b65f80d825b02db3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:42:50.528146 2417 raft_consensus.cc:2454] T e6f202f8f2ad456386d9bb5d5e3070c5 P 1b9d2814da9c4fe89992a72a0bbb858c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b745a811fae54cd996e42b4fc6f14864 in term 1.
I20260928 06:42:50.528316 2283 raft_consensus.cc:2454] T e6f202f8f2ad456386d9bb5d5e3070c5 P 5dd26eed1e5243c7b65f80d825b02db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b745a811fae54cd996e42b4fc6f14864 in term 1.
I20260928 06:42:50.528342 2489 leader_election.cc:302] T e6f202f8f2ad456386d9bb5d5e3070c5 P b745a811fae54cd996e42b4fc6f14864 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b9d2814da9c4fe89992a72a0bbb858c, b745a811fae54cd996e42b4fc6f14864; no voters:
I20260928 06:42:50.528424 2624 raft_consensus.cc:2781] T e6f202f8f2ad456386d9bb5d5e3070c5 P b745a811fae54cd996e42b4fc6f14864 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:42:50.528577 2624 raft_consensus.cc:686] T e6f202f8f2ad456386d9bb5d5e3070c5 P b745a811fae54cd996e42b4fc6f14864 [term 1 LEADER]: Becoming Leader. State: Replica: b745a811fae54cd996e42b4fc6f14864, State: Running, Role: LEADER
I20260928 06:42:50.528678 2624 consensus_queue.cc:237] T e6f202f8f2ad456386d9bb5d5e3070c5 P b745a811fae54cd996e42b4fc6f14864 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b745a811fae54cd996e42b4fc6f14864" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40893 } } peers { permanent_uuid: "5dd26eed1e5243c7b65f80d825b02db3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 40593 } } peers { permanent_uuid: "1b9d2814da9c4fe89992a72a0bbb858c" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33735 } }
I20260928 06:42:50.529605 2136 catalog_manager.cc:5495] T e6f202f8f2ad456386d9bb5d5e3070c5 P b745a811fae54cd996e42b4fc6f14864 reported cstate change: term changed from 0 to 1, leader changed from <none> to b745a811fae54cd996e42b4fc6f14864 (127.18.211.67). New cstate: current_term: 1 leader_uuid: "b745a811fae54cd996e42b4fc6f14864" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b745a811fae54cd996e42b4fc6f14864" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5dd26eed1e5243c7b65f80d825b02db3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 40593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b9d2814da9c4fe89992a72a0bbb858c" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33735 } health_report { overall_health: UNKNOWN } } }
I20260928 06:42:50.562439 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:50.560136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53804 (local address 127.18.211.67:40893)
0928 06:42:50.560209 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 06:42:50.560211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:42:50.560234 (+ 23us) server_negotiation.cc:407] Connection header received
0928 06:42:50.560321 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:42:50.560322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:42:50.560342 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:42:50.560403 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:42:50.560515 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:50.560980 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:50.561772 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:42:50.561972 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:42:50.562026 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:42:50.562142 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:42:50.562251 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:42:50.562307 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:42:50.562344 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 06:42:50.562362 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 06:42:50.564404 2417 raft_consensus.cc:1261] T e6f202f8f2ad456386d9bb5d5e3070c5 P 1b9d2814da9c4fe89992a72a0bbb858c [term 1 FOLLOWER]: Refusing update from remote peer b745a811fae54cd996e42b4fc6f14864: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:42:50.564417 2283 raft_consensus.cc:1261] T e6f202f8f2ad456386d9bb5d5e3070c5 P 5dd26eed1e5243c7b65f80d825b02db3 [term 1 FOLLOWER]: Refusing update from remote peer b745a811fae54cd996e42b4fc6f14864: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:42:50.564752 2624 consensus_queue.cc:1035] T e6f202f8f2ad456386d9bb5d5e3070c5 P b745a811fae54cd996e42b4fc6f14864 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd26eed1e5243c7b65f80d825b02db3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 40593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:42:50.564934 2628 consensus_queue.cc:1035] T e6f202f8f2ad456386d9bb5d5e3070c5 P b745a811fae54cd996e42b4fc6f14864 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b9d2814da9c4fe89992a72a0bbb858c" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:42:50.567477 2631 mvcc.cc:205] Tried to move back new op lower bound from 7334206548229095424 to 7334206548086063104. Current Snapshot: MvccSnapshot[applied={T|T < 7334206548229095424}]
I20260928 06:42:50.567822 2632 mvcc.cc:205] Tried to move back new op lower bound from 7334206548229095424 to 7334206548086063104. Current Snapshot: MvccSnapshot[applied={T|T < 7334206548229095424}]
I20260928 06:42:50.567960 2633 mvcc.cc:205] Tried to move back new op lower bound from 7334206548229095424 to 7334206548086063104. Current Snapshot: MvccSnapshot[applied={T|T < 7334206548229095424 or (T in {7334206548229095424})}]
I20260928 06:42:50.570326 2137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:42:50.570410 2137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:42:50.571138 2137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98: Sending DeleteTablet for 3 replicas of tablet e6f202f8f2ad456386d9bb5d5e3070c5
I20260928 06:42:50.571439 2534 tablet_service.cc:1514] Processing DeleteTablet for tablet e6f202f8f2ad456386d9bb5d5e3070c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:42:50 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:53798
I20260928 06:42:50.571483 2263 tablet_service.cc:1514] Processing DeleteTablet for tablet e6f202f8f2ad456386d9bb5d5e3070c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:42:50 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:34190
I20260928 06:42:50.571486 2397 tablet_service.cc:1514] Processing DeleteTablet for tablet e6f202f8f2ad456386d9bb5d5e3070c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:42:50 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:35038
I20260928 06:42:50.571755 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 2200
I20260928 06:42:50.571810 2643 tablet_replica.cc:330] T e6f202f8f2ad456386d9bb5d5e3070c5 P 1b9d2814da9c4fe89992a72a0bbb858c: stopping tablet replica
I20260928 06:42:50.571898 2643 raft_consensus.cc:2229] T e6f202f8f2ad456386d9bb5d5e3070c5 P 1b9d2814da9c4fe89992a72a0bbb858c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:42:50.571990 2643 raft_consensus.cc:2258] T e6f202f8f2ad456386d9bb5d5e3070c5 P 1b9d2814da9c4fe89992a72a0bbb858c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:42:50.572121 2641 tablet_replica.cc:330] T e6f202f8f2ad456386d9bb5d5e3070c5 P b745a811fae54cd996e42b4fc6f14864: stopping tablet replica
I20260928 06:42:50.572196 2641 raft_consensus.cc:2229] T e6f202f8f2ad456386d9bb5d5e3070c5 P b745a811fae54cd996e42b4fc6f14864 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:42:50.572305 2641 raft_consensus.cc:2258] T e6f202f8f2ad456386d9bb5d5e3070c5 P b745a811fae54cd996e42b4fc6f14864 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:42:50.572777 2641 ts_tablet_manager.cc:1905] T e6f202f8f2ad456386d9bb5d5e3070c5 P b745a811fae54cd996e42b4fc6f14864: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:42:50.573060 2643 ts_tablet_manager.cc:1905] T e6f202f8f2ad456386d9bb5d5e3070c5 P 1b9d2814da9c4fe89992a72a0bbb858c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:42:50.574368 2641 ts_tablet_manager.cc:1918] T e6f202f8f2ad456386d9bb5d5e3070c5 P b745a811fae54cd996e42b4fc6f14864: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:42:50.574455 2641 log.cc:1196] T e6f202f8f2ad456386d9bb5d5e3070c5 P b745a811fae54cd996e42b4fc6f14864: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/wal/wals/e6f202f8f2ad456386d9bb5d5e3070c5
I20260928 06:42:50.574733 2641 ts_tablet_manager.cc:1939] T e6f202f8f2ad456386d9bb5d5e3070c5 P b745a811fae54cd996e42b4fc6f14864: Deleting consensus metadata
I20260928 06:42:50.574936 2643 ts_tablet_manager.cc:1918] T e6f202f8f2ad456386d9bb5d5e3070c5 P 1b9d2814da9c4fe89992a72a0bbb858c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:42:50.575171 2643 log.cc:1196] T e6f202f8f2ad456386d9bb5d5e3070c5 P 1b9d2814da9c4fe89992a72a0bbb858c: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/wal/wals/e6f202f8f2ad456386d9bb5d5e3070c5
I20260928 06:42:50.575138 2124 catalog_manager.cc:4841] TS b745a811fae54cd996e42b4fc6f14864 (127.18.211.67:40893): tablet e6f202f8f2ad456386d9bb5d5e3070c5 (table test-table [id=1e82901d5b8f46a1aa88d670eb22d624]) successfully deleted
I20260928 06:42:50.575536 2643 ts_tablet_manager.cc:1939] T e6f202f8f2ad456386d9bb5d5e3070c5 P 1b9d2814da9c4fe89992a72a0bbb858c: Deleting consensus metadata
I20260928 06:42:50.576050 2124 catalog_manager.cc:4841] TS 1b9d2814da9c4fe89992a72a0bbb858c (127.18.211.66:33735): tablet e6f202f8f2ad456386d9bb5d5e3070c5 (table test-table [id=1e82901d5b8f46a1aa88d670eb22d624]) successfully deleted
W20260928 06:42:50.578151 2123 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.211.65:40593 (error 108)
I20260928 06:42:50.578436 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 2334
I20260928 06:42:50.579054 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:42:50.578538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:40593 (local address 127.0.0.1:34202)
0928 06:42:50.578979 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 06:42:50.579020 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.65:40593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420}
I20260928 06:42:50.582957 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 2470
W20260928 06:42:50.582957 2123 catalog_manager.cc:4584] TS 5dd26eed1e5243c7b65f80d825b02db3 (127.18.211.65:40593): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e6f202f8f2ad456386d9bb5d5e3070c5: Network error: Client connection negotiation failed: client connection to 127.18.211.65:40593: connect: Connection refused (error 111)
I20260928 06:42:50.587251 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 2105
I20260928 06:43:10.592008 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:35839
--webserver_interface=127.18.211.126
--webserver_port=40407
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:35839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:43:10.672659 2645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:43:10.672880 2645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:43:10.672904 2645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:43:10.674077 2645 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 06:43:10.674126 2645 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 06:43:10.674139 2645 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 06:43:10.674149 2645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:43:10.674163 2645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:43:10.674188 2645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:43:10.674206 2645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:43:10.674227 2645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:43:10.675791 2645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:35839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:35839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=40407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:43:10.676036 2645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:43:10.676276 2645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:43:10.684870 2651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:10.684907 2652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:10.684996 2645 server_base.cc:1034] running on GCE node
W20260928 06:43:10.684902 2654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:10.685444 2645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:43:10.685774 2645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:43:10.686936 2645 hybrid_clock.cc:648] HybridClock initialized: now 1790577790686921 us; error 31 us; skew 500 ppm
Sep 28 06:43:10 dist-test-slave-b28r krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577790, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:43:10.689476 2645 init.cc:377] Logged in from keytab as kudu/127.18.211.126@KRBTEST.COM (short username kudu)
I20260928 06:43:10.689945 2645 webserver.cc:460] Webserver started at http://127.18.211.126:40407/ using document root <none> and password file <none>
I20260928 06:43:10.690138 2645 fs_manager.cc:359] Metadata directory not provided
I20260928 06:43:10.690189 2645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:43:10.691617 2645 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:43:10.692205 2661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:10.692396 2645 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:10.692476 2645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "f0cbe448c598422385926ffb3fef2e98"
format_stamp: "Formatted at 2026-09-28 06:42:49 on dist-test-slave-b28r"
server_key: "4b5b1a3974a136b611475cfff92cc3ae"
server_key_iv: "5ad50cc53ae0c0032e5abbdae82f41c6"
server_key_version: "encryptionkey@0"
I20260928 06:43:10.692862 2645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:43:10.704391 2645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:43:10.706516 2645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:43:10.706629 2645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:43:10.710470 2645 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:35839
I20260928 06:43:10.710517 2713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:35839 every 8 connection(s)
I20260928 06:43:10.710878 2645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:43:10.712365 2714 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:43:10.713025 2714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98: Bootstrap starting.
I20260928 06:43:10.717062 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 2645
I20260928 06:43:10.717334 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.65:40593
--local_ip_for_outbound_sockets=127.18.211.65
--tserver_master_addrs=127.18.211.126:35839
--webserver_port=33261
--webserver_interface=127.18.211.65
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:43:10.721926 2714 log.cc:824] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98: Log is configured to *not* fsync() on all Append() calls
I20260928 06:43:10.723976 2714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:43:10.724256 2714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98: Bootstrap complete.
I20260928 06:43:10.726378 2714 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0cbe448c598422385926ffb3fef2e98" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35839 } }
I20260928 06:43:10.726606 2714 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0cbe448c598422385926ffb3fef2e98, State: Initialized, Role: FOLLOWER
I20260928 06:43:10.726760 2714 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0cbe448c598422385926ffb3fef2e98" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35839 } }
I20260928 06:43:10.726835 2714 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:43:10.726890 2714 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:43:10.726940 2714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [term 1 FOLLOWER]: Advancing to term 2
I20260928 06:43:10.727990 2714 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0cbe448c598422385926ffb3fef2e98" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35839 } }
I20260928 06:43:10.728085 2714 leader_election.cc:302] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0cbe448c598422385926ffb3fef2e98; no voters:
I20260928 06:43:10.728243 2714 leader_election.cc:288] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 06:43:10.728294 2718 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 06:43:10.728458 2718 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [term 2 LEADER]: Becoming Leader. State: Replica: f0cbe448c598422385926ffb3fef2e98, State: Running, Role: LEADER
I20260928 06:43:10.728519 2714 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:43:10.728576 2718 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0cbe448c598422385926ffb3fef2e98" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35839 } }
I20260928 06:43:10.728904 2719 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f0cbe448c598422385926ffb3fef2e98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0cbe448c598422385926ffb3fef2e98" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35839 } } }
I20260928 06:43:10.728992 2719 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [sys.catalog]: This master's current role is: LEADER
I20260928 06:43:10.729212 2720 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0cbe448c598422385926ffb3fef2e98. Latest consensus state: current_term: 2 leader_uuid: "f0cbe448c598422385926ffb3fef2e98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0cbe448c598422385926ffb3fef2e98" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35839 } } }
I20260928 06:43:10.729275 2720 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [sys.catalog]: This master's current role is: LEADER
I20260928 06:43:10.730262 2734 catalog_manager.cc:1259] Loaded cluster ID: a098db18aaa54e6d84d051b51772f69a
I20260928 06:43:10.730312 2734 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98: loading cluster ID for follower catalog manager: success
I20260928 06:43:10.731478 2734 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98: acquiring CA information for follower catalog manager: success
I20260928 06:43:10.731885 2734 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:43:10.731957 2724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:43:10.732374 2724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:43:10.732446 2724 catalog_manager.cc:1259] Loaded cluster ID: a098db18aaa54e6d84d051b51772f69a
I20260928 06:43:10.732467 2724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:43:10.732630 2724 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:43:10.732703 2724 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98: Loaded TSK: 0
I20260928 06:43:10.741403 2724 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98: Generated new TSK 1
I20260928 06:43:10.741488 2724 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:43:10.798192 2716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:43:10.798434 2716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:43:10.798466 2716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:43:10.799690 2716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:43:10.799733 2716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:43:10.799768 2716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.65
I20260928 06:43:10.801223 2716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.65:40593
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.211.65
--webserver_port=33261
--tserver_master_addrs=127.18.211.126:35839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.65
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:43:10.801457 2716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:43:10.801662 2716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:43:10.807401 2743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:10.807428 2745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:10.807431 2742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:10.807650 2716 server_base.cc:1034] running on GCE node
I20260928 06:43:10.807842 2716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:43:10.808068 2716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:43:10.809242 2716 hybrid_clock.cc:648] HybridClock initialized: now 1790577790809226 us; error 38 us; skew 500 ppm
Sep 28 06:43:10 dist-test-slave-b28r krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577790, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:43:10.811703 2716 init.cc:377] Logged in from keytab as kudu/127.18.211.65@KRBTEST.COM (short username kudu)
I20260928 06:43:10.812144 2716 webserver.cc:460] Webserver started at http://127.18.211.65:33261/ using document root <none> and password file <none>
I20260928 06:43:10.812391 2716 fs_manager.cc:359] Metadata directory not provided
I20260928 06:43:10.812454 2716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:43:10.813798 2716 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:43:10.814471 2752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:10.814625 2716 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:43:10.814706 2716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/wal
uuid: "5dd26eed1e5243c7b65f80d825b02db3"
format_stamp: "Formatted at 2026-09-28 06:42:50 on dist-test-slave-b28r"
server_key: "71f571bee70f0fc6d587479b21c5e2fb"
server_key_iv: "c8e57d9b01fc0c7b30b19354ed03964f"
server_key_version: "encryptionkey@0"
I20260928 06:43:10.815027 2716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:43:10.824258 2716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:43:10.826221 2716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:43:10.826357 2716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:43:10.826562 2716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:43:10.826972 2759 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 06:43:10.828951 2716 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 06:43:10.829002 2716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 06:43:10.829035 2716 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 06:43:10.829718 2716 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 06:43:10.829751 2716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 06:43:10.829838 2759 tablet_bootstrap.cc:492] T e6f202f8f2ad456386d9bb5d5e3070c5 P 5dd26eed1e5243c7b65f80d825b02db3: Bootstrap starting.
I20260928 06:43:10.836705 2716 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.65:40593
I20260928 06:43:10.837060 2716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
I20260928 06:43:10.839331 2866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.65:40593 every 8 connection(s)
I20260928 06:43:10.841101 2759 log.cc:824] T e6f202f8f2ad456386d9bb5d5e3070c5 P 5dd26eed1e5243c7b65f80d825b02db3: Log is configured to *not* fsync() on all Append() calls
I20260928 06:43:10.842485 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 2716
I20260928 06:43:10.842779 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.66:33735
--local_ip_for_outbound_sockets=127.18.211.66
--tserver_master_addrs=127.18.211.126:35839
--webserver_port=33867
--webserver_interface=127.18.211.66
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:43:10.843888 2759 tablet_bootstrap.cc:492] T e6f202f8f2ad456386d9bb5d5e3070c5 P 5dd26eed1e5243c7b65f80d825b02db3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:43:10.844198 2759 tablet_bootstrap.cc:492] T e6f202f8f2ad456386d9bb5d5e3070c5 P 5dd26eed1e5243c7b65f80d825b02db3: Bootstrap complete.
I20260928 06:43:10.845381 2759 ts_tablet_manager.cc:1397] T e6f202f8f2ad456386d9bb5d5e3070c5 P 5dd26eed1e5243c7b65f80d825b02db3: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.003s
I20260928 06:43:10.846740 2759 raft_consensus.cc:348] T e6f202f8f2ad456386d9bb5d5e3070c5 P 5dd26eed1e5243c7b65f80d825b02db3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b745a811fae54cd996e42b4fc6f14864" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40893 } } peers { permanent_uuid: "5dd26eed1e5243c7b65f80d825b02db3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 40593 } } peers { permanent_uuid: "1b9d2814da9c4fe89992a72a0bbb858c" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33735 } }
I20260928 06:43:10.847071 2759 raft_consensus.cc:729] T e6f202f8f2ad456386d9bb5d5e3070c5 P 5dd26eed1e5243c7b65f80d825b02db3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd26eed1e5243c7b65f80d825b02db3, State: Initialized, Role: FOLLOWER
I20260928 06:43:10.847262 2759 consensus_queue.cc:260] T e6f202f8f2ad456386d9bb5d5e3070c5 P 5dd26eed1e5243c7b65f80d825b02db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b745a811fae54cd996e42b4fc6f14864" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40893 } } peers { permanent_uuid: "5dd26eed1e5243c7b65f80d825b02db3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 40593 } } peers { permanent_uuid: "1b9d2814da9c4fe89992a72a0bbb858c" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33735 } }
I20260928 06:43:10.847548 2759 ts_tablet_manager.cc:1428] T e6f202f8f2ad456386d9bb5d5e3070c5 P 5dd26eed1e5243c7b65f80d825b02db3: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
Sep 28 06:43:10 dist-test-slave-b28r krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577790, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:43:10.853837 2869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:10.837673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.65:36035)
0928 06:43:10.841302 (+ 3629us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:10.841356 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:10.841906 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:10.842696 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:10.842700 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:10.843325 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:43:10.843607 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:10.843614 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:10.845080 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:10.845081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:10.846329 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:10.846333 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:10.847248 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:10.847248 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:10.847395 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:10.848072 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:43:10.848083 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:43:10.849821 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:43:10.852024 (+ 2203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:43:10.852030 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:43:10.852032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:43:10.852332 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:43:10.852622 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:43:10.852623 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:43:10.852624 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:43:10.852750 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:43:10.853204 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:43:10.853210 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:43:10.853366 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 06:43:10.853465 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 06:43:10.853636 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3578,"thread_start_us":108,"threads_started":1}
I20260928 06:43:10.854272 2870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:10.837768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:36035 (local address 127.18.211.126:35839)
0928 06:43:10.842115 (+ 4347us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:10.842122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:10.842152 (+ 30us) server_negotiation.cc:407] Connection header received
0928 06:43:10.842283 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:10.842290 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:10.842353 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:10.842514 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:10.843797 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:10.844976 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:10.846425 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:10.847023 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:10.847126 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:10.849947 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:10.849952 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:10.849952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:43:10.849966 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:10.851932 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:10.852427 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:10.852430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:10.852431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:10.852554 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:10.852823 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:10.852825 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:10.852825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:10.853129 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:10.853216 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:10.853556 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 06:43:10.853711 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4297,"thread_start_us":73,"threads_started":1}
I20260928 06:43:10.855448 2867 heartbeater.cc:347] Connected to a master server at 127.18.211.126:35839
I20260928 06:43:10.855537 2867 heartbeater.cc:464] Registering TS with master...
I20260928 06:43:10.855700 2867 heartbeater.cc:511] Master 127.18.211.126:35839 requested a full tablet report, sending...
I20260928 06:43:10.856411 2678 ts_manager.cc:194] Registered new tserver with Master: 5dd26eed1e5243c7b65f80d825b02db3 (127.18.211.65:40593)
I20260928 06:43:10.857637 2678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.65@KRBTEST.COM'} at 127.18.211.65:36035
I20260928 06:43:10.858567 2867 heartbeater.cc:503] Master 127.18.211.126:35839 was elected leader, sending a full tablet report...
Sep 28 06:43:10 dist-test-slave-b28r krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577790, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for kudu/127.18.211.65@KRBTEST.COM
I20260928 06:43:10.867238 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:10.857010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:40593 (local address 127.0.0.1:57038)
0928 06:43:10.857480 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:10.857519 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:10.857588 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:10.858085 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:10.858089 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:10.858405 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:43:10.858459 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:10.858462 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:10.859996 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:10.859997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:10.860929 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:10.860933 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:10.861567 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:10.861568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:10.861649 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:10.862153 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:43:10.862158 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:43:10.863790 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:43:10.865737 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:43:10.865744 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:43:10.865746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:43:10.866106 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:43:10.866460 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:43:10.866461 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:43:10.866461 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:43:10.866566 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:43:10.866898 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:43:10.866905 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:43:10.866952 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 06:43:10.867028 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 06:43:10.867164 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":102,"threads_started":1}
I20260928 06:43:10.867281 2875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:10.857302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57038 (local address 127.18.211.65:40593)
0928 06:43:10.857720 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:10.857726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:10.857736 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:43:10.857767 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:10.857769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:10.857781 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:10.857837 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:10.858550 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:10.859916 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:10.861018 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:10.861491 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:10.861553 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:10.863881 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:10.863889 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:10.863890 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:43:10.863896 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:10.865586 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:10.866240 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:10.866243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:10.866244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:10.866347 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:10.866620 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:10.866621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:10.866622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:10.866804 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:10.866895 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:10.867073 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 06:43:10.867188 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":87,"threads_started":1}
I20260928 06:43:10.867672 2788 tablet_service.cc:1514] Processing DeleteTablet for tablet e6f202f8f2ad456386d9bb5d5e3070c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:42:50 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:57038
I20260928 06:43:10.868284 2876 tablet_replica.cc:330] stopping tablet replica
I20260928 06:43:10.868394 2876 raft_consensus.cc:2229] T e6f202f8f2ad456386d9bb5d5e3070c5 P 5dd26eed1e5243c7b65f80d825b02db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:43:10.868460 2876 raft_consensus.cc:2258] T e6f202f8f2ad456386d9bb5d5e3070c5 P 5dd26eed1e5243c7b65f80d825b02db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:43:10.868729 2876 ts_tablet_manager.cc:1905] T e6f202f8f2ad456386d9bb5d5e3070c5 P 5dd26eed1e5243c7b65f80d825b02db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:43:10.870656 2876 ts_tablet_manager.cc:1918] T e6f202f8f2ad456386d9bb5d5e3070c5 P 5dd26eed1e5243c7b65f80d825b02db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:43:10.870721 2876 log.cc:1196] T e6f202f8f2ad456386d9bb5d5e3070c5 P 5dd26eed1e5243c7b65f80d825b02db3: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/wal/wals/e6f202f8f2ad456386d9bb5d5e3070c5
I20260928 06:43:10.870900 2876 ts_tablet_manager.cc:1939] T e6f202f8f2ad456386d9bb5d5e3070c5 P 5dd26eed1e5243c7b65f80d825b02db3: Deleting consensus metadata
I20260928 06:43:10.871467 2664 catalog_manager.cc:4841] TS 5dd26eed1e5243c7b65f80d825b02db3 (127.18.211.65:40593): tablet e6f202f8f2ad456386d9bb5d5e3070c5 (table test-table [id=1e82901d5b8f46a1aa88d670eb22d624]) successfully deleted
W20260928 06:43:10.931062 2872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:43:10.931265 2872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:43:10.931288 2872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:43:10.932430 2872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:43:10.932479 2872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:43:10.932505 2872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.66
I20260928 06:43:10.933952 2872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.66:33735
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.211.66
--webserver_port=33867
--tserver_master_addrs=127.18.211.126:35839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.66
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:43:10.934160 2872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:43:10.934360 2872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:43:10.939044 2884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:10.939134 2882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:10.939107 2881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:10.939158 2872 server_base.cc:1034] running on GCE node
I20260928 06:43:10.939360 2872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:43:10.939566 2872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:43:10.940704 2872 hybrid_clock.cc:648] HybridClock initialized: now 1790577790940689 us; error 31 us; skew 500 ppm
Sep 28 06:43:10 dist-test-slave-b28r krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577790, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:43:10.942977 2872 init.cc:377] Logged in from keytab as kudu/127.18.211.66@KRBTEST.COM (short username kudu)
I20260928 06:43:10.943419 2872 webserver.cc:460] Webserver started at http://127.18.211.66:33867/ using document root <none> and password file <none>
I20260928 06:43:10.943641 2872 fs_manager.cc:359] Metadata directory not provided
I20260928 06:43:10.943703 2872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:43:10.944926 2872 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:43:10.945429 2892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:10.945576 2872 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:10.945653 2872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/wal
uuid: "1b9d2814da9c4fe89992a72a0bbb858c"
format_stamp: "Formatted at 2026-09-28 06:42:50 on dist-test-slave-b28r"
server_key: "62ed72ee70ef531d283b0d048a7d09bd"
server_key_iv: "86648977cc16e351de47fc7474884780"
server_key_version: "encryptionkey@0"
I20260928 06:43:10.945952 2872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:43:10.956632 2872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:43:10.958295 2872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:43:10.958419 2872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:43:10.958616 2872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:43:10.958889 2872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:43:10.958921 2872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:10.958946 2872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:43:10.958972 2872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:10.964527 2872 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.66:33735
I20260928 06:43:10.964582 3005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.66:33735 every 8 connection(s)
I20260928 06:43:10.964875 2872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
I20260928 06:43:10.970206 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 2872
I20260928 06:43:10.970453 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.67:40893
--local_ip_for_outbound_sockets=127.18.211.67
--tserver_master_addrs=127.18.211.126:35839
--webserver_port=40923
--webserver_interface=127.18.211.67
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 06:43:10 dist-test-slave-b28r krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577790, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:43:10.973598 2870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:10.965506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:34577 (local address 127.18.211.126:35839)
0928 06:43:10.965548 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:10.965550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:10.965746 (+ 196us) server_negotiation.cc:407] Connection header received
0928 06:43:10.966235 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:10.966236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:10.966253 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:10.966335 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:10.967277 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:10.967768 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:10.968753 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:10.968956 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:10.969011 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:10.971044 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:10.971046 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:10.971046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:43:10.971053 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:10.972387 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:10.972737 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:10.972738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:10.972739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:10.972796 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:10.973004 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:10.973005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:10.973005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:10.973116 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:10.973170 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:10.973420 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 06:43:10.973507 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:43:10.973738 3008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:10.965484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.66:34577)
0928 06:43:10.965642 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:10.965674 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:10.966144 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:10.966417 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:10.966421 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:10.966956 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:43:10.967186 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:10.967192 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:10.967850 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:10.967851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:10.968672 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:10.968675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:10.969025 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:10.969026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:10.969153 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:10.969599 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:43:10.969607 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:43:10.970957 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:43:10.972461 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:43:10.972468 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:43:10.972470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:43:10.972673 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:43:10.972848 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:43:10.972849 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:43:10.972849 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:43:10.972950 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:43:10.973167 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:43:10.973174 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:43:10.973306 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 06:43:10.973392 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 06:43:10.973540 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
I20260928 06:43:10.974330 3006 heartbeater.cc:347] Connected to a master server at 127.18.211.126:35839
I20260928 06:43:10.974462 3006 heartbeater.cc:464] Registering TS with master...
I20260928 06:43:10.974651 3006 heartbeater.cc:511] Master 127.18.211.126:35839 requested a full tablet report, sending...
I20260928 06:43:10.975050 2678 ts_manager.cc:194] Registered new tserver with Master: 1b9d2814da9c4fe89992a72a0bbb858c (127.18.211.66:33735)
I20260928 06:43:10.975426 2678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.66@KRBTEST.COM'} at 127.18.211.66:34577
W20260928 06:43:11.050076 3009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:43:11.050275 3009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:43:11.050305 3009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:43:11.051424 3009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:43:11.051460 3009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:43:11.051496 3009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.67
I20260928 06:43:11.052878 3009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.67:40893
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.211.67
--webserver_port=40923
--tserver_master_addrs=127.18.211.126:35839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.67
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:43:11.053093 3009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:43:11.053301 3009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:43:11.057973 3015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:11.058030 3009 server_base.cc:1034] running on GCE node
W20260928 06:43:11.058007 3014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:11.057981 3017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:11.058394 3009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:43:11.058584 3009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:43:11.059734 3009 hybrid_clock.cc:648] HybridClock initialized: now 1790577791059715 us; error 33 us; skew 500 ppm
Sep 28 06:43:11 dist-test-slave-b28r krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577791, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:43:11.062049 3009 init.cc:377] Logged in from keytab as kudu/127.18.211.67@KRBTEST.COM (short username kudu)
I20260928 06:43:11.062486 3009 webserver.cc:460] Webserver started at http://127.18.211.67:40923/ using document root <none> and password file <none>
I20260928 06:43:11.062669 3009 fs_manager.cc:359] Metadata directory not provided
I20260928 06:43:11.062737 3009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:43:11.064061 3009 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:43:11.064615 3025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:11.064777 3009 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:43:11.064860 3009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/wal
uuid: "b745a811fae54cd996e42b4fc6f14864"
format_stamp: "Formatted at 2026-09-28 06:42:50 on dist-test-slave-b28r"
server_key: "a11ae81d480488b6272d098a3c4f6043"
server_key_iv: "e99f696f5a2faebcacebde03a8571912"
server_key_version: "encryptionkey@0"
I20260928 06:43:11.065189 3009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:43:11.075968 3009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:43:11.077780 3009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:43:11.077910 3009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:43:11.078109 3009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:43:11.078390 3009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:43:11.078423 3009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:11.078449 3009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:43:11.078467 3009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:11.084079 3009 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.67:40893
I20260928 06:43:11.084136 3138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.67:40893 every 8 connection(s)
I20260928 06:43:11.084450 3009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
I20260928 06:43:11.085100 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 3009
Sep 28 06:43:11 dist-test-slave-b28r krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577791, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:43:11.091709 3142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:11.085494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48850 (local address 127.18.211.126:35839)
0928 06:43:11.085591 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:11.085593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:11.085695 (+ 102us) server_negotiation.cc:407] Connection header received
0928 06:43:11.085768 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:11.085768 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:11.085780 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:11.085842 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:11.086476 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:11.086980 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:11.087730 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:11.087960 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:11.088015 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:11.089193 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:11.089195 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:11.089195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:43:11.089201 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:11.090620 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:11.090927 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:11.090929 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:11.090929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:11.091018 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:11.091206 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:11.091207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:11.091207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:11.091313 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:11.091471 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:11.091534 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 06:43:11.091615 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":46,"threads_started":1}
I20260928 06:43:11.093398 2870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:11.085135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:43031 (local address 127.18.211.126:35839)
0928 06:43:11.085195 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:11.085197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:11.085455 (+ 258us) server_negotiation.cc:407] Connection header received
0928 06:43:11.085975 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:11.085975 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:11.085983 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:11.086029 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:11.086924 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:11.087438 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:11.088387 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:11.088576 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:11.088627 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:11.090907 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:11.090908 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:11.090908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:43:11.090914 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:11.092291 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:11.092650 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:11.092651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:11.092651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:11.092698 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:11.092910 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:11.092912 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:11.092912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:11.093020 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:11.093082 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:11.093285 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 06:43:11.093342 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 06:43:11.093608 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:11.085145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.67:43031)
0928 06:43:11.085319 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:11.085367 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:11.085899 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:11.086081 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:11.086084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:11.086631 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:43:11.086823 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:11.086830 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:11.087518 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:11.087519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:11.088304 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:11.088306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:11.088641 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:11.088642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:11.088759 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:11.089185 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:43:11.089194 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:43:11.090833 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:43:11.092364 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:43:11.092371 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:43:11.092373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:43:11.092594 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:43:11.092745 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:43:11.092746 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:43:11.092746 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:43:11.092833 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:43:11.093081 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:43:11.093085 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:43:11.093194 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 06:43:11.093271 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 06:43:11.093410 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":92,"threads_started":1}
I20260928 06:43:11.094003 3139 heartbeater.cc:347] Connected to a master server at 127.18.211.126:35839
I20260928 06:43:11.094087 3139 heartbeater.cc:464] Registering TS with master...
I20260928 06:43:11.094228 3139 heartbeater.cc:511] Master 127.18.211.126:35839 requested a full tablet report, sending...
I20260928 06:43:11.094558 2678 ts_manager.cc:194] Registered new tserver with Master: b745a811fae54cd996e42b4fc6f14864 (127.18.211.67:40893)
I20260928 06:43:11.094918 2678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.67@KRBTEST.COM'} at 127.18.211.67:43031
I20260928 06:43:11.103274 19277 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:43:11.106002 2870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:11.103616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48856 (local address 127.18.211.126:35839)
0928 06:43:11.103715 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:11.103717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:11.103834 (+ 117us) server_negotiation.cc:407] Connection header received
0928 06:43:11.103904 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:11.103905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:11.103944 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:11.104003 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:43:11.104141 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:11.104611 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:11.105345 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:11.105525 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:11.105578 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:11.105768 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:43:11.105810 (+ 42us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 06:43:11.105939 (+ 129us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48856: token expired
Metrics: {"server-negotiator.queue_time_us":70}
W20260928 06:43:11.106062 2870 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48856: token expired
W20260928 06:43:11.106168 2607 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.18.211.126:35839)
I20260928 06:43:11.111645 2870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:11.106431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48862 (local address 127.18.211.126:35839)
0928 06:43:11.106484 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:11.106486 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:11.106494 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:43:11.106513 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:11.106513 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:11.106521 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:11.106565 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:11.107014 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:11.107520 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:11.108186 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:11.108377 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:11.108429 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:11.109500 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:11.109502 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:11.109502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:43:11.109509 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:11.110651 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:11.110964 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:11.110965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:11.110965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:11.111021 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:11.111212 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:11.111213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:11.111213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:11.111314 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:11.111420 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:11.111493 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 06:43:11.111562 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 06:43:11.112730 2678 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:43:11.114329 2678 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:43:11.116484 2788 tablet_service.cc:1467] Processing CreateTablet for tablet 936fc1451c584bad8d4b4ca64390c57a (DEFAULT_TABLE table=test-table [id=c132a97a7dec45bbb616e84efc76758f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:43:11.116663 2788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 936fc1451c584bad8d4b4ca64390c57a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:43:11.118068 2759 tablet_bootstrap.cc:492] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3: Bootstrap starting.
I20260928 06:43:11.118414 2759 tablet_bootstrap.cc:654] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3: Neither blocks nor log segments found. Creating new log.
I20260928 06:43:11.118996 2759 tablet_bootstrap.cc:492] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3: No bootstrap required, opened a new log
I20260928 06:43:11.119048 2759 ts_tablet_manager.cc:1397] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260928 06:43:11.119429 2759 raft_consensus.cc:348] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b9d2814da9c4fe89992a72a0bbb858c" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33735 } } peers { permanent_uuid: "b745a811fae54cd996e42b4fc6f14864" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40893 } } peers { permanent_uuid: "5dd26eed1e5243c7b65f80d825b02db3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 40593 } }
I20260928 06:43:11.119508 2759 raft_consensus.cc:374] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:43:11.119550 2759 raft_consensus.cc:729] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd26eed1e5243c7b65f80d825b02db3, State: Initialized, Role: FOLLOWER
I20260928 06:43:11.119592 2759 consensus_queue.cc:260] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b9d2814da9c4fe89992a72a0bbb858c" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33735 } } peers { permanent_uuid: "b745a811fae54cd996e42b4fc6f14864" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40893 } } peers { permanent_uuid: "5dd26eed1e5243c7b65f80d825b02db3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 40593 } }
I20260928 06:43:11.119683 2759 ts_tablet_manager.cc:1428] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 06:43:11.120148 3145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:11.116176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:40893 (local address 127.0.0.1:46658)
0928 06:43:11.116344 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:11.116354 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:11.116396 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:11.116750 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:11.116751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:11.116753 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:43:11.116799 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:11.116801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:11.117849 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:11.117850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:11.118920 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:11.118923 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:11.119894 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:11.119895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:11.119955 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:11.119975 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 06:43:11.120030 (+ 55us) client_negotiation.cc:240] Negotiation successful
0928 06:43:11.120071 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":102,"threads_started":1}
I20260928 06:43:11.120160 3147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:11.116423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46658 (local address 127.18.211.67:40893)
0928 06:43:11.116586 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:11.116591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:11.116602 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:43:11.116641 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:11.116644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:11.116658 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:11.116739 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:43:11.116883 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:11.117779 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:11.119062 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:11.119816 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:11.119885 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:11.119947 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:11.120047 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 06:43:11.120083 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":87,"threads_started":1}
I20260928 06:43:11.120929 3073 tablet_service.cc:1467] Processing CreateTablet for tablet 936fc1451c584bad8d4b4ca64390c57a (DEFAULT_TABLE table=test-table [id=c132a97a7dec45bbb616e84efc76758f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:43:11.121265 3073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 936fc1451c584bad8d4b4ca64390c57a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:43:11.121335 3146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:11.116212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56592 (local address 127.18.211.66:33735)
0928 06:43:11.118137 (+ 1925us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:11.118142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:11.118154 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:43:11.118198 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:11.118201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:11.118218 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:11.118322 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:43:11.118423 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:11.119446 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:11.120565 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:11.121022 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:11.121075 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:11.121110 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:11.121246 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 06:43:11.121273 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1880,"thread_start_us":76,"threads_started":1}
I20260928 06:43:11.121342 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:11.116090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:33735 (local address 127.0.0.1:56592)
0928 06:43:11.116204 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:11.116213 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:11.116248 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:11.118322 (+ 2074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:11.118323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:11.118324 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:43:11.118366 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:11.118368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:11.119530 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:11.119531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:11.120501 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:11.120505 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:11.121096 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:11.121097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:11.121170 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:11.121174 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:43:11.121238 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 06:43:11.121278 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":10}
I20260928 06:43:11.122046 2940 tablet_service.cc:1467] Processing CreateTablet for tablet 936fc1451c584bad8d4b4ca64390c57a (DEFAULT_TABLE table=test-table [id=c132a97a7dec45bbb616e84efc76758f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:43:11.122395 2940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 936fc1451c584bad8d4b4ca64390c57a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:43:11.124753 3149 tablet_bootstrap.cc:492] T 936fc1451c584bad8d4b4ca64390c57a P b745a811fae54cd996e42b4fc6f14864: Bootstrap starting.
I20260928 06:43:11.125349 3149 tablet_bootstrap.cc:654] T 936fc1451c584bad8d4b4ca64390c57a P b745a811fae54cd996e42b4fc6f14864: Neither blocks nor log segments found. Creating new log.
I20260928 06:43:11.125556 3150 tablet_bootstrap.cc:492] T 936fc1451c584bad8d4b4ca64390c57a P 1b9d2814da9c4fe89992a72a0bbb858c: Bootstrap starting.
I20260928 06:43:11.125870 3149 log.cc:824] T 936fc1451c584bad8d4b4ca64390c57a P b745a811fae54cd996e42b4fc6f14864: Log is configured to *not* fsync() on all Append() calls
I20260928 06:43:11.126106 3150 tablet_bootstrap.cc:654] T 936fc1451c584bad8d4b4ca64390c57a P 1b9d2814da9c4fe89992a72a0bbb858c: Neither blocks nor log segments found. Creating new log.
I20260928 06:43:11.126574 3149 tablet_bootstrap.cc:492] T 936fc1451c584bad8d4b4ca64390c57a P b745a811fae54cd996e42b4fc6f14864: No bootstrap required, opened a new log
I20260928 06:43:11.126600 3150 log.cc:824] T 936fc1451c584bad8d4b4ca64390c57a P 1b9d2814da9c4fe89992a72a0bbb858c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:43:11.126636 3149 ts_tablet_manager.cc:1397] T 936fc1451c584bad8d4b4ca64390c57a P b745a811fae54cd996e42b4fc6f14864: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:43:11.127274 3150 tablet_bootstrap.cc:492] T 936fc1451c584bad8d4b4ca64390c57a P 1b9d2814da9c4fe89992a72a0bbb858c: No bootstrap required, opened a new log
I20260928 06:43:11.127324 3150 ts_tablet_manager.cc:1397] T 936fc1451c584bad8d4b4ca64390c57a P 1b9d2814da9c4fe89992a72a0bbb858c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:43:11.128273 3149 raft_consensus.cc:348] T 936fc1451c584bad8d4b4ca64390c57a P b745a811fae54cd996e42b4fc6f14864 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b9d2814da9c4fe89992a72a0bbb858c" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33735 } } peers { permanent_uuid: "b745a811fae54cd996e42b4fc6f14864" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40893 } } peers { permanent_uuid: "5dd26eed1e5243c7b65f80d825b02db3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 40593 } }
I20260928 06:43:11.128420 3149 raft_consensus.cc:374] T 936fc1451c584bad8d4b4ca64390c57a P b745a811fae54cd996e42b4fc6f14864 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:43:11.128464 3149 raft_consensus.cc:729] T 936fc1451c584bad8d4b4ca64390c57a P b745a811fae54cd996e42b4fc6f14864 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b745a811fae54cd996e42b4fc6f14864, State: Initialized, Role: FOLLOWER
I20260928 06:43:11.128577 3149 consensus_queue.cc:260] T 936fc1451c584bad8d4b4ca64390c57a P b745a811fae54cd996e42b4fc6f14864 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b9d2814da9c4fe89992a72a0bbb858c" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33735 } } peers { permanent_uuid: "b745a811fae54cd996e42b4fc6f14864" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40893 } } peers { permanent_uuid: "5dd26eed1e5243c7b65f80d825b02db3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 40593 } }
I20260928 06:43:11.128820 3149 ts_tablet_manager.cc:1428] T 936fc1451c584bad8d4b4ca64390c57a P b745a811fae54cd996e42b4fc6f14864: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:43:11.128953 3139 heartbeater.cc:503] Master 127.18.211.126:35839 was elected leader, sending a full tablet report...
I20260928 06:43:11.128978 3150 raft_consensus.cc:348] T 936fc1451c584bad8d4b4ca64390c57a P 1b9d2814da9c4fe89992a72a0bbb858c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b9d2814da9c4fe89992a72a0bbb858c" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33735 } } peers { permanent_uuid: "b745a811fae54cd996e42b4fc6f14864" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40893 } } peers { permanent_uuid: "5dd26eed1e5243c7b65f80d825b02db3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 40593 } }
I20260928 06:43:11.129124 3150 raft_consensus.cc:374] T 936fc1451c584bad8d4b4ca64390c57a P 1b9d2814da9c4fe89992a72a0bbb858c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:43:11.129148 3150 raft_consensus.cc:729] T 936fc1451c584bad8d4b4ca64390c57a P 1b9d2814da9c4fe89992a72a0bbb858c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b9d2814da9c4fe89992a72a0bbb858c, State: Initialized, Role: FOLLOWER
I20260928 06:43:11.129253 3150 consensus_queue.cc:260] T 936fc1451c584bad8d4b4ca64390c57a P 1b9d2814da9c4fe89992a72a0bbb858c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b9d2814da9c4fe89992a72a0bbb858c" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33735 } } peers { permanent_uuid: "b745a811fae54cd996e42b4fc6f14864" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40893 } } peers { permanent_uuid: "5dd26eed1e5243c7b65f80d825b02db3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 40593 } }
I20260928 06:43:11.129563 3150 ts_tablet_manager.cc:1428] T 936fc1451c584bad8d4b4ca64390c57a P 1b9d2814da9c4fe89992a72a0bbb858c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:43:11.129597 3006 heartbeater.cc:503] Master 127.18.211.126:35839 was elected leader, sending a full tablet report...
W20260928 06:43:11.215423 3007 tablet.cc:2369] T 936fc1451c584bad8d4b4ca64390c57a P 1b9d2814da9c4fe89992a72a0bbb858c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:43:11.255326 2873 raft_consensus.cc:482] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:43:11.255465 2873 raft_consensus.cc:504] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b9d2814da9c4fe89992a72a0bbb858c" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33735 } } peers { permanent_uuid: "b745a811fae54cd996e42b4fc6f14864" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40893 } } peers { permanent_uuid: "5dd26eed1e5243c7b65f80d825b02db3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 40593 } }
I20260928 06:43:11.255761 2873 leader_election.cc:288] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b9d2814da9c4fe89992a72a0bbb858c (127.18.211.66:33735), b745a811fae54cd996e42b4fc6f14864 (127.18.211.67:40893)
I20260928 06:43:11.258728 3147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:11.256140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:42941 (local address 127.18.211.67:40893)
0928 06:43:11.256192 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:11.256194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:11.256291 (+ 97us) server_negotiation.cc:407] Connection header received
0928 06:43:11.256322 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:11.256323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:11.256333 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:11.256399 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:43:11.256544 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:11.257048 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:11.258002 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:11.258376 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:11.258437 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:11.258446 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:11.258622 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 06:43:11.258661 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 06:43:11.258733 3146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:11.256078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:55049 (local address 127.18.211.66:33735)
0928 06:43:11.256132 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:11.256134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:11.256170 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:43:11.256201 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:11.256202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:11.256212 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:11.256265 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:43:11.256431 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:11.256968 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:11.257947 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:11.258379 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:11.258446 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:11.258454 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:11.258625 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 06:43:11.258663 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 06:43:11.258775 2869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:11.255947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:33735 (local address 127.18.211.65:55049)
0928 06:43:11.256021 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:11.256034 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:11.256116 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:11.256299 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:11.256300 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:11.256303 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:43:11.256365 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:11.256366 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:11.257040 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:11.257040 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:11.257857 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:11.257859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:11.258455 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:11.258456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:11.258538 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:11.258559 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 06:43:11.258608 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 06:43:11.258650 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":2}
I20260928 06:43:11.258795 3155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:11.256036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:40893 (local address 127.18.211.65:42941)
0928 06:43:11.256197 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:11.256206 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:11.256272 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:11.256405 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:11.256406 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:11.256407 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:43:11.256481 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:11.256483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:11.257121 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:11.257122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:11.257910 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:11.257912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:11.258458 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:11.258459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:11.258533 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:11.258555 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 06:43:11.258609 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 06:43:11.258656 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":3,"thread_start_us":95,"threads_started":1}
I20260928 06:43:11.259210 3093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "936fc1451c584bad8d4b4ca64390c57a" candidate_uuid: "5dd26eed1e5243c7b65f80d825b02db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b745a811fae54cd996e42b4fc6f14864" is_pre_election: true
I20260928 06:43:11.259196 2960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "936fc1451c584bad8d4b4ca64390c57a" candidate_uuid: "5dd26eed1e5243c7b65f80d825b02db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b9d2814da9c4fe89992a72a0bbb858c" is_pre_election: true
I20260928 06:43:11.259346 2960 raft_consensus.cc:2454] T 936fc1451c584bad8d4b4ca64390c57a P 1b9d2814da9c4fe89992a72a0bbb858c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd26eed1e5243c7b65f80d825b02db3 in term 0.
I20260928 06:43:11.259352 3093 raft_consensus.cc:2454] T 936fc1451c584bad8d4b4ca64390c57a P b745a811fae54cd996e42b4fc6f14864 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd26eed1e5243c7b65f80d825b02db3 in term 0.
I20260928 06:43:11.259567 2756 leader_election.cc:302] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd26eed1e5243c7b65f80d825b02db3, b745a811fae54cd996e42b4fc6f14864; no voters:
I20260928 06:43:11.259683 2873 raft_consensus.cc:2781] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:43:11.259753 2873 raft_consensus.cc:482] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:43:11.259778 2873 raft_consensus.cc:3037] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:43:11.260535 2873 raft_consensus.cc:504] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b9d2814da9c4fe89992a72a0bbb858c" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33735 } } peers { permanent_uuid: "b745a811fae54cd996e42b4fc6f14864" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40893 } } peers { permanent_uuid: "5dd26eed1e5243c7b65f80d825b02db3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 40593 } }
I20260928 06:43:11.260660 2873 leader_election.cc:288] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [CANDIDATE]: Term 1 election: Requested vote from peers 1b9d2814da9c4fe89992a72a0bbb858c (127.18.211.66:33735), b745a811fae54cd996e42b4fc6f14864 (127.18.211.67:40893)
I20260928 06:43:11.260874 2960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "936fc1451c584bad8d4b4ca64390c57a" candidate_uuid: "5dd26eed1e5243c7b65f80d825b02db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b9d2814da9c4fe89992a72a0bbb858c"
I20260928 06:43:11.260895 3093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "936fc1451c584bad8d4b4ca64390c57a" candidate_uuid: "5dd26eed1e5243c7b65f80d825b02db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b745a811fae54cd996e42b4fc6f14864"
I20260928 06:43:11.260952 2960 raft_consensus.cc:3037] T 936fc1451c584bad8d4b4ca64390c57a P 1b9d2814da9c4fe89992a72a0bbb858c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:43:11.260962 3093 raft_consensus.cc:3037] T 936fc1451c584bad8d4b4ca64390c57a P b745a811fae54cd996e42b4fc6f14864 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:43:11.261632 3093 raft_consensus.cc:2454] T 936fc1451c584bad8d4b4ca64390c57a P b745a811fae54cd996e42b4fc6f14864 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd26eed1e5243c7b65f80d825b02db3 in term 1.
I20260928 06:43:11.261714 2960 raft_consensus.cc:2454] T 936fc1451c584bad8d4b4ca64390c57a P 1b9d2814da9c4fe89992a72a0bbb858c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd26eed1e5243c7b65f80d825b02db3 in term 1.
I20260928 06:43:11.261794 2756 leader_election.cc:302] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5dd26eed1e5243c7b65f80d825b02db3, b745a811fae54cd996e42b4fc6f14864; no voters:
I20260928 06:43:11.261870 2873 raft_consensus.cc:2781] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:43:11.262081 2873 raft_consensus.cc:686] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [term 1 LEADER]: Becoming Leader. State: Replica: 5dd26eed1e5243c7b65f80d825b02db3, State: Running, Role: LEADER
I20260928 06:43:11.262148 2873 consensus_queue.cc:237] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b9d2814da9c4fe89992a72a0bbb858c" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33735 } } peers { permanent_uuid: "b745a811fae54cd996e42b4fc6f14864" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40893 } } peers { permanent_uuid: "5dd26eed1e5243c7b65f80d825b02db3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 40593 } }
I20260928 06:43:11.262940 2677 catalog_manager.cc:5495] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5dd26eed1e5243c7b65f80d825b02db3 (127.18.211.65). New cstate: current_term: 1 leader_uuid: "5dd26eed1e5243c7b65f80d825b02db3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b9d2814da9c4fe89992a72a0bbb858c" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b745a811fae54cd996e42b4fc6f14864" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5dd26eed1e5243c7b65f80d825b02db3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 40593 } health_report { overall_health: HEALTHY } } }
I20260928 06:43:11.304211 2875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:11.301867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57044 (local address 127.18.211.65:40593)
0928 06:43:11.301920 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:11.301922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:11.301933 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:43:11.302001 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:11.302002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:11.302023 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:11.302082 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:43:11.302218 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:11.302744 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:11.303524 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:11.303742 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:11.303793 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:11.303902 (+ 109us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:43:11.304009 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:43:11.304062 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:11.304109 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 06:43:11.304132 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 06:43:11.305855 2960 raft_consensus.cc:1261] T 936fc1451c584bad8d4b4ca64390c57a P 1b9d2814da9c4fe89992a72a0bbb858c [term 1 FOLLOWER]: Refusing update from remote peer 5dd26eed1e5243c7b65f80d825b02db3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:43:11.305855 3093 raft_consensus.cc:1261] T 936fc1451c584bad8d4b4ca64390c57a P b745a811fae54cd996e42b4fc6f14864 [term 1 FOLLOWER]: Refusing update from remote peer 5dd26eed1e5243c7b65f80d825b02db3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:43:11.306212 3156 consensus_queue.cc:1035] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b745a811fae54cd996e42b4fc6f14864" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:43:11.306377 2873 consensus_queue.cc:1035] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b9d2814da9c4fe89992a72a0bbb858c" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:43:11.308753 3159 mvcc.cc:205] Tried to move back new op lower bound from 7334206633185935360 to 7334206633010266112. Current Snapshot: MvccSnapshot[applied={T|T < 7334206633185935360}]
I20260928 06:43:11.309062 3161 mvcc.cc:205] Tried to move back new op lower bound from 7334206633185935360 to 7334206633010266112. Current Snapshot: MvccSnapshot[applied={T|T < 7334206633185935360 or (T in {7334206633185935360})}]
I20260928 06:43:11.311007 2677 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:43:11.311131 2677 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:43:11.311841 2677 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98: Sending DeleteTablet for 3 replicas of tablet 936fc1451c584bad8d4b4ca64390c57a
I20260928 06:43:11.312115 2788 tablet_service.cc:1514] Processing DeleteTablet for tablet 936fc1451c584bad8d4b4ca64390c57a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:43:11 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:57038
I20260928 06:43:11.312151 3073 tablet_service.cc:1514] Processing DeleteTablet for tablet 936fc1451c584bad8d4b4ca64390c57a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:43:11 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:46658
I20260928 06:43:11.312263 2940 tablet_service.cc:1514] Processing DeleteTablet for tablet 936fc1451c584bad8d4b4ca64390c57a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:43:11 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:56592
I20260928 06:43:11.312470 2876 tablet_replica.cc:330] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3: stopping tablet replica
I20260928 06:43:11.312493 3169 tablet_replica.cc:330] T 936fc1451c584bad8d4b4ca64390c57a P 1b9d2814da9c4fe89992a72a0bbb858c: stopping tablet replica
I20260928 06:43:11.312539 2876 raft_consensus.cc:2229] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:43:11.312589 3169 raft_consensus.cc:2229] T 936fc1451c584bad8d4b4ca64390c57a P 1b9d2814da9c4fe89992a72a0bbb858c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:43:11.312620 2876 raft_consensus.cc:2258] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:43:11.312700 3169 raft_consensus.cc:2258] T 936fc1451c584bad8d4b4ca64390c57a P 1b9d2814da9c4fe89992a72a0bbb858c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:43:11.312940 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 2716
I20260928 06:43:11.313023 2876 ts_tablet_manager.cc:1905] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:43:11.313555 3169 ts_tablet_manager.cc:1905] T 936fc1451c584bad8d4b4ca64390c57a P 1b9d2814da9c4fe89992a72a0bbb858c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:43:11.314860 3169 ts_tablet_manager.cc:1918] T 936fc1451c584bad8d4b4ca64390c57a P 1b9d2814da9c4fe89992a72a0bbb858c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:43:11.314926 3169 log.cc:1196] T 936fc1451c584bad8d4b4ca64390c57a P 1b9d2814da9c4fe89992a72a0bbb858c: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/wal/wals/936fc1451c584bad8d4b4ca64390c57a
I20260928 06:43:11.315207 3169 ts_tablet_manager.cc:1939] T 936fc1451c584bad8d4b4ca64390c57a P 1b9d2814da9c4fe89992a72a0bbb858c: Deleting consensus metadata
I20260928 06:43:11.315619 2665 catalog_manager.cc:4841] TS 1b9d2814da9c4fe89992a72a0bbb858c (127.18.211.66:33735): tablet 936fc1451c584bad8d4b4ca64390c57a (table test-table [id=c132a97a7dec45bbb616e84efc76758f]) successfully deleted
W20260928 06:43:11.318061 2664 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.211.65:40593 (error 108)
I20260928 06:43:11.318418 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 2872
I20260928 06:43:11.318418 3168 tablet_replica.cc:330] T 936fc1451c584bad8d4b4ca64390c57a P b745a811fae54cd996e42b4fc6f14864: stopping tablet replica
I20260928 06:43:11.318570 2874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:11.318463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:40593 (local address 127.0.0.1:57050)
0928 06:43:11.318509 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:11.318543 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.65:40593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 06:43:11.319808 3168 raft_consensus.cc:2229] T 936fc1451c584bad8d4b4ca64390c57a P b745a811fae54cd996e42b4fc6f14864 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:43:11.319958 3168 raft_consensus.cc:2258] T 936fc1451c584bad8d4b4ca64390c57a P b745a811fae54cd996e42b4fc6f14864 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:43:11.320485 3168 ts_tablet_manager.cc:1905] T 936fc1451c584bad8d4b4ca64390c57a P b745a811fae54cd996e42b4fc6f14864: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 06:43:11.320595 2664 catalog_manager.cc:4584] TS 5dd26eed1e5243c7b65f80d825b02db3 (127.18.211.65:40593): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 936fc1451c584bad8d4b4ca64390c57a: Network error: Client connection negotiation failed: client connection to 127.18.211.65:40593: connect: Connection refused (error 111)
I20260928 06:43:11.321760 3168 ts_tablet_manager.cc:1918] T 936fc1451c584bad8d4b4ca64390c57a P b745a811fae54cd996e42b4fc6f14864: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:43:11.321822 3168 log.cc:1196] T 936fc1451c584bad8d4b4ca64390c57a P b745a811fae54cd996e42b4fc6f14864: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/wal/wals/936fc1451c584bad8d4b4ca64390c57a
I20260928 06:43:11.322036 3168 ts_tablet_manager.cc:1939] T 936fc1451c584bad8d4b4ca64390c57a P b745a811fae54cd996e42b4fc6f14864: Deleting consensus metadata
I20260928 06:43:11.322449 2665 catalog_manager.cc:4841] TS b745a811fae54cd996e42b4fc6f14864 (127.18.211.67:40893): tablet 936fc1451c584bad8d4b4ca64390c57a (table test-table [id=c132a97a7dec45bbb616e84efc76758f]) successfully deleted
I20260928 06:43:11.323885 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 3009
I20260928 06:43:11.327981 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 2645
I20260928 06:43:31.332762 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:35839
--webserver_interface=127.18.211.126
--webserver_port=40407
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:35839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:43:31.411610 3171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:43:31.411829 3171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:43:31.411850 3171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:43:31.412982 3171 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 06:43:31.413022 3171 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 06:43:31.413033 3171 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 06:43:31.413043 3171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:43:31.413064 3171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:43:31.413089 3171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:43:31.413111 3171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:43:31.413133 3171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:43:31.414587 3171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:35839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:35839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=40407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:43:31.414820 3171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:43:31.415027 3171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:43:31.419818 3179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:31.419824 3177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:31.419857 3176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:31.419852 3171 server_base.cc:1034] running on GCE node
I20260928 06:43:31.420306 3171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:43:31.420570 3171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:43:31.421711 3171 hybrid_clock.cc:648] HybridClock initialized: now 1790577811421699 us; error 30 us; skew 500 ppm
Sep 28 06:43:31 dist-test-slave-b28r krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577811, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:43:31.424046 3171 init.cc:377] Logged in from keytab as kudu/127.18.211.126@KRBTEST.COM (short username kudu)
I20260928 06:43:31.424467 3171 webserver.cc:460] Webserver started at http://127.18.211.126:40407/ using document root <none> and password file <none>
I20260928 06:43:31.424692 3171 fs_manager.cc:359] Metadata directory not provided
I20260928 06:43:31.424741 3171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:43:31.426049 3171 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:43:31.426692 3186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:31.426851 3171 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:43:31.426923 3171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "f0cbe448c598422385926ffb3fef2e98"
format_stamp: "Formatted at 2026-09-28 06:42:49 on dist-test-slave-b28r"
server_key: "4b5b1a3974a136b611475cfff92cc3ae"
server_key_iv: "5ad50cc53ae0c0032e5abbdae82f41c6"
server_key_version: "encryptionkey@0"
I20260928 06:43:31.427306 3171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:43:31.442929 3171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:43:31.444685 3171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:43:31.444824 3171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:43:31.448810 3171 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:35839
I20260928 06:43:31.448850 3238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:35839 every 8 connection(s)
I20260928 06:43:31.449196 3171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:43:31.450634 3239 sys_catalog.cc:263] Verifying existing consensus state
I20260928 06:43:31.451280 3239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98: Bootstrap starting.
I20260928 06:43:31.457619 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 3171
I20260928 06:43:31.457955 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.65:40593
--local_ip_for_outbound_sockets=127.18.211.65
--tserver_master_addrs=127.18.211.126:35839
--webserver_port=33261
--webserver_interface=127.18.211.65
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:43:31.460088 3239 log.cc:824] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98: Log is configured to *not* fsync() on all Append() calls
I20260928 06:43:31.462289 3239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:43:31.462561 3239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98: Bootstrap complete.
I20260928 06:43:31.464632 3239 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0cbe448c598422385926ffb3fef2e98" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35839 } }
I20260928 06:43:31.464854 3239 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0cbe448c598422385926ffb3fef2e98, State: Initialized, Role: FOLLOWER
I20260928 06:43:31.465013 3239 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0cbe448c598422385926ffb3fef2e98" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35839 } }
I20260928 06:43:31.465090 3239 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:43:31.465137 3239 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:43:31.465183 3239 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [term 2 FOLLOWER]: Advancing to term 3
I20260928 06:43:31.466120 3239 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0cbe448c598422385926ffb3fef2e98" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35839 } }
I20260928 06:43:31.466226 3239 leader_election.cc:302] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0cbe448c598422385926ffb3fef2e98; no voters:
I20260928 06:43:31.466378 3239 leader_election.cc:288] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 06:43:31.466454 3244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 06:43:31.466626 3239 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:43:31.466678 3244 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [term 3 LEADER]: Becoming Leader. State: Replica: f0cbe448c598422385926ffb3fef2e98, State: Running, Role: LEADER
I20260928 06:43:31.466840 3244 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0cbe448c598422385926ffb3fef2e98" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35839 } }
I20260928 06:43:31.467149 3245 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f0cbe448c598422385926ffb3fef2e98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0cbe448c598422385926ffb3fef2e98" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35839 } } }
I20260928 06:43:31.467236 3245 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [sys.catalog]: This master's current role is: LEADER
I20260928 06:43:31.467468 3246 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0cbe448c598422385926ffb3fef2e98. Latest consensus state: current_term: 3 leader_uuid: "f0cbe448c598422385926ffb3fef2e98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0cbe448c598422385926ffb3fef2e98" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35839 } } }
I20260928 06:43:31.467537 3246 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98 [sys.catalog]: This master's current role is: LEADER
I20260928 06:43:31.467533 3253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:43:31.468397 3253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:43:31.468492 3253 catalog_manager.cc:1259] Loaded cluster ID: a098db18aaa54e6d84d051b51772f69a
I20260928 06:43:31.468524 3253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:43:31.469290 3253 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:43:31.469450 3253 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98: Loaded TSK: 0
I20260928 06:43:31.469487 3253 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f0cbe448c598422385926ffb3fef2e98: Loaded TSK: 1
F20260928 06:43:31.469687 3253 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd40bc139ad google::LogMessage::Fail() at ??:0
@ 0x7fd40bc1688a google::LogMessage::SendToLog() at ??:0
@ 0x7fd40bc134de google::LogMessage::Flush() at ??:0
@ 0x7fd40bc14a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd40e380104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fd40e37dd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fd413b6d1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fd413b6d48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fd413b6d594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fd413b33c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fd413b3dad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd40dae10d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd40dad6954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd40e93a6db start_thread at ??:0
@ 0x7fd40a71161f clone at ??:0
W20260928 06:43:31.553606 3241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:43:31.553807 3241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:43:31.553839 3241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:43:31.554965 3241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:43:31.555007 3241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:43:31.555040 3241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.65
I20260928 06:43:31.556497 3241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.65:40593
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.211.65
--webserver_port=33261
--tserver_master_addrs=127.18.211.126:35839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.65
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:43:31.556725 3241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:43:31.556928 3241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:43:31.562402 3282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:31.562397 3285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:31.562649 3241 server_base.cc:1034] running on GCE node
W20260928 06:43:31.562404 3283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:31.562920 3241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:43:31.563185 3241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:43:31.564342 3241 hybrid_clock.cc:648] HybridClock initialized: now 1790577811564329 us; error 33 us; skew 500 ppm
Sep 28 06:43:31 dist-test-slave-b28r krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577811, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:43:31.566804 3241 init.cc:377] Logged in from keytab as kudu/127.18.211.65@KRBTEST.COM (short username kudu)
I20260928 06:43:31.567322 3241 webserver.cc:460] Webserver started at http://127.18.211.65:33261/ using document root <none> and password file <none>
I20260928 06:43:31.567526 3241 fs_manager.cc:359] Metadata directory not provided
I20260928 06:43:31.567576 3241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:43:31.568894 3241 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:43:31.569626 3292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:31.569823 3241 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:43:31.569908 3241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/wal
uuid: "5dd26eed1e5243c7b65f80d825b02db3"
format_stamp: "Formatted at 2026-09-28 06:42:50 on dist-test-slave-b28r"
server_key: "71f571bee70f0fc6d587479b21c5e2fb"
server_key_iv: "c8e57d9b01fc0c7b30b19354ed03964f"
server_key_version: "encryptionkey@0"
I20260928 06:43:31.570297 3241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:43:31.587590 3241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:43:31.589720 3241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:43:31.589854 3241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:43:31.590067 3241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:43:31.590519 3299 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 06:43:31.592514 3241 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 06:43:31.592571 3241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260928 06:43:31.592605 3241 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 06:43:31.593331 3241 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 06:43:31.593369 3241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260928 06:43:31.593464 3299 tablet_bootstrap.cc:492] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3: Bootstrap starting.
I20260928 06:43:31.595284 3299 log.cc:824] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3: Log is configured to *not* fsync() on all Append() calls
I20260928 06:43:31.598193 3299 tablet_bootstrap.cc:492] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 06:43:31.598507 3299 tablet_bootstrap.cc:492] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3: Bootstrap complete.
I20260928 06:43:31.598656 3299 ts_tablet_manager.cc:1397] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260928 06:43:31.600026 3241 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.65:40593
I20260928 06:43:31.600349 3241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
I20260928 06:43:31.600834 3408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.65:40593 every 8 connection(s)
I20260928 06:43:31.600548 3299 raft_consensus.cc:348] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b9d2814da9c4fe89992a72a0bbb858c" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33735 } } peers { permanent_uuid: "b745a811fae54cd996e42b4fc6f14864" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40893 } } peers { permanent_uuid: "5dd26eed1e5243c7b65f80d825b02db3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 40593 } }
I20260928 06:43:31.600996 3299 raft_consensus.cc:729] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd26eed1e5243c7b65f80d825b02db3, State: Initialized, Role: FOLLOWER
I20260928 06:43:31.601121 3299 consensus_queue.cc:260] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b9d2814da9c4fe89992a72a0bbb858c" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33735 } } peers { permanent_uuid: "b745a811fae54cd996e42b4fc6f14864" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40893 } } peers { permanent_uuid: "5dd26eed1e5243c7b65f80d825b02db3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 40593 } }
I20260928 06:43:31.601387 3299 ts_tablet_manager.cc:1428] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260928 06:43:31.603226 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 3241
I20260928 06:43:31.603457 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.66:33735
--local_ip_for_outbound_sockets=127.18.211.66
--tserver_master_addrs=127.18.211.126:35839
--webserver_port=33867
--webserver_interface=127.18.211.66
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:43:31.681035 3411 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:43:31.601003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.65:55279)
0928 06:43:31.601199 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:31.601246 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:31.601405 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:31.680827 (+ 79422us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":100,"threads_started":1}
W20260928 06:43:31.681304 3409 heartbeater.cc:644] Failed to heartbeat to 127.18.211.126:35839 (0 consecutive failures): Network error: Failed to ping master at 127.18.211.126:35839: Client connection negotiation failed: client connection to 127.18.211.126:35839: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 06:43:31.681641 3411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:31.681531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.65:52247)
0928 06:43:31.681578 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:31.681619 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 06:43:31.681931 3411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:31.681849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.65:36029)
0928 06:43:31.681887 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:31.681909 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 06:43:31.682036 3409 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 06:43:31.708031 3413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:43:31.708237 3413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:43:31.708261 3413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:43:31.709391 3413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:43:31.709431 3413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:43:31.709457 3413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.66
I20260928 06:43:31.710840 3413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.66:33735
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.211.66
--webserver_port=33867
--tserver_master_addrs=127.18.211.126:35839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.66
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:43:31.711020 3413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:43:31.711247 3413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:43:31.715821 3422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:31.715852 3419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:31.715821 3420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:31.716215 3413 server_base.cc:1034] running on GCE node
I20260928 06:43:31.716427 3413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:43:31.716647 3413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:43:31.717813 3413 hybrid_clock.cc:648] HybridClock initialized: now 1790577811717800 us; error 30 us; skew 500 ppm
Sep 28 06:43:31 dist-test-slave-b28r krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577811, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:43:31.720455 3413 init.cc:377] Logged in from keytab as kudu/127.18.211.66@KRBTEST.COM (short username kudu)
I20260928 06:43:31.720893 3413 webserver.cc:460] Webserver started at http://127.18.211.66:33867/ using document root <none> and password file <none>
I20260928 06:43:31.721091 3413 fs_manager.cc:359] Metadata directory not provided
I20260928 06:43:31.721138 3413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:43:31.722554 3413 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:43:31.723223 3429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:31.723412 3413 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:43:31.723495 3413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/wal
uuid: "1b9d2814da9c4fe89992a72a0bbb858c"
format_stamp: "Formatted at 2026-09-28 06:42:50 on dist-test-slave-b28r"
server_key: "62ed72ee70ef531d283b0d048a7d09bd"
server_key_iv: "86648977cc16e351de47fc7474884780"
server_key_version: "encryptionkey@0"
I20260928 06:43:31.723840 3413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:43:31.732438 3413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:43:31.734325 3413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:43:31.734455 3413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:43:31.734669 3413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:43:31.734984 3413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:43:31.735023 3413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:31.735050 3413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:43:31.735103 3413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:31.740705 3413 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.66:33735
I20260928 06:43:31.740777 3542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.66:33735 every 8 connection(s)
I20260928 06:43:31.741086 3413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
I20260928 06:43:31.742189 3545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:31.741736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.66:53573)
0928 06:43:31.741934 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:31.742009 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":83,"threads_started":1}
W20260928 06:43:31.742611 3543 heartbeater.cc:644] Failed to heartbeat to 127.18.211.126:35839 (0 consecutive failures): Network error: Failed to ping master at 127.18.211.126:35839: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
I20260928 06:43:31.742874 3545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:31.742790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.66:47251)
0928 06:43:31.742831 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:31.742858 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 06:43:31.743068 3545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:31.743021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.66:33737)
0928 06:43:31.743041 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:31.743057 (+ 16us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260928 06:43:31.743165 3543 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 06:43:31.749081 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 3413
I20260928 06:43:31.749356 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.67:40893
--local_ip_for_outbound_sockets=127.18.211.67
--tserver_master_addrs=127.18.211.126:35839
--webserver_port=40923
--webserver_interface=127.18.211.67
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:40003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:43:31.829506 3546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:43:31.829710 3546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:43:31.829732 3546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:43:31.830852 3546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:43:31.830885 3546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:43:31.830911 3546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.67
I20260928 06:43:31.832350 3546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:40003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.67:40893
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.211.67
--webserver_port=40923
--tserver_master_addrs=127.18.211.126:35839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.67
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:43:31.832531 3546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:43:31.832705 3546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:43:31.837702 3546 server_base.cc:1034] running on GCE node
W20260928 06:43:31.837644 3551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:31.837663 3554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:31.837679 3552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:31.838070 3546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:43:31.838294 3546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:43:31.839452 3546 hybrid_clock.cc:648] HybridClock initialized: now 1790577811839439 us; error 28 us; skew 500 ppm
Sep 28 06:43:31 dist-test-slave-b28r krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577811, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:43:31.841893 3546 init.cc:377] Logged in from keytab as kudu/127.18.211.67@KRBTEST.COM (short username kudu)
I20260928 06:43:31.842432 3546 webserver.cc:460] Webserver started at http://127.18.211.67:40923/ using document root <none> and password file <none>
I20260928 06:43:31.842639 3546 fs_manager.cc:359] Metadata directory not provided
I20260928 06:43:31.842689 3546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:43:31.844074 3546 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:43:31.844707 3562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:31.844878 3546 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:43:31.844960 3546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/wal
uuid: "b745a811fae54cd996e42b4fc6f14864"
format_stamp: "Formatted at 2026-09-28 06:42:50 on dist-test-slave-b28r"
server_key: "a11ae81d480488b6272d098a3c4f6043"
server_key_iv: "e99f696f5a2faebcacebde03a8571912"
server_key_version: "encryptionkey@0"
I20260928 06:43:31.845343 3546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:43:31.852608 3546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:43:31.854553 3546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:43:31.854688 3546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:43:31.854894 3546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:43:31.855237 3546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:43:31.855271 3546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:31.855300 3546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:43:31.855317 3546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:31.861343 3546 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.67:40893
I20260928 06:43:31.861409 3675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.67:40893 every 8 connection(s)
I20260928 06:43:31.861698 3546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
I20260928 06:43:31.862697 3678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:31.862338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.67:37045)
0928 06:43:31.862492 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:31.862560 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
W20260928 06:43:31.863052 3676 heartbeater.cc:644] Failed to heartbeat to 127.18.211.126:35839 (0 consecutive failures): Network error: Failed to ping master at 127.18.211.126:35839: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
I20260928 06:43:31.863348 3678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:31.863259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.67:52871)
0928 06:43:31.863298 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:31.863331 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 06:43:31.863540 3678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:31.863493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.67:59905)
0928 06:43:31.863513 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:31.863530 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260928 06:43:31.863618 3676 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 06:43:31.863857 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 3546
I20260928 06:43:32.682709 3680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:32.682462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.65:49419)
0928 06:43:32.682604 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:32.682642 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":64,"threads_started":1}
I20260928 06:43:32.743667 3681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:32.743475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.66:54929)
0928 06:43:32.743581 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:32.743616 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":55,"threads_started":1}
I20260928 06:43:32.864089 3682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:32.863887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.67:34639)
0928 06:43:32.864013 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:32.864040 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
I20260928 06:43:32.990228 3683 raft_consensus.cc:482] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:43:32.990361 3683 raft_consensus.cc:504] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b9d2814da9c4fe89992a72a0bbb858c" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33735 } } peers { permanent_uuid: "b745a811fae54cd996e42b4fc6f14864" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40893 } } peers { permanent_uuid: "5dd26eed1e5243c7b65f80d825b02db3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 40593 } }
I20260928 06:43:32.990665 3683 leader_election.cc:288] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b9d2814da9c4fe89992a72a0bbb858c (127.18.211.66:33735), b745a811fae54cd996e42b4fc6f14864 (127.18.211.67:40893)
Sep 28 06:43:32 dist-test-slave-b28r krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577811, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for kudu/127.18.211.67@KRBTEST.COM
Sep 28 06:43:32 dist-test-slave-b28r krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577811, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for kudu/127.18.211.66@KRBTEST.COM
I20260928 06:43:33.000474 3685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:32.990946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:44823 (local address 127.18.211.66:33735)
0928 06:43:32.991100 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:32.991114 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:32.991152 (+ 38us) server_negotiation.cc:407] Connection header received
0928 06:43:32.991668 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:32.991671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:32.991712 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:32.991828 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:32.992785 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:32.993762 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:32.994532 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:32.994907 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:32.995002 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:32.997546 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:32.997551 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:32.997551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:43:32.997563 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:32.999047 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:32.999414 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:32.999417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:32.999417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:32.999521 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:32.999699 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:32.999700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:32.999701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:32.999927 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:32.999994 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:33.000241 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 06:43:33.000357 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":66,"threads_started":1}
I20260928 06:43:33.000479 3686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:32.991144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:33931 (local address 127.18.211.67:40893)
0928 06:43:32.991304 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:32.991314 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:32.991342 (+ 28us) server_negotiation.cc:407] Connection header received
0928 06:43:32.991740 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:32.991742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:32.991778 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:32.991912 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:32.992604 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:32.993530 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:32.994522 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:32.994907 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:32.995002 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:32.997146 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:32.997150 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:32.997150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:43:32.997160 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:32.998767 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:32.999168 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:32.999170 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:32.999171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:32.999278 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:32.999665 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:32.999666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:32.999666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:32.999902 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:32.999972 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:33.000249 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 06:43:33.000373 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
I20260928 06:43:33.000497 3680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:32.990822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:33735 (local address 127.18.211.65:44823)
0928 06:43:32.990881 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:32.990894 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:32.990965 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:32.991905 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:32.991908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:32.992305 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:43:32.992682 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:32.992685 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:32.993854 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:32.993854 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:32.994456 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:32.994458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:32.994992 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:32.994993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:32.995133 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:32.995674 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:43:32.995678 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:43:32.997438 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:43:32.999178 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:43:32.999179 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:43:32.999179 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:43:32.999342 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:43:32.999592 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:43:32.999593 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:43:32.999593 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:43:32.999647 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:43:32.999989 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:43:32.999990 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:43:33.000093 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 06:43:33.000172 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 06:43:33.000359 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":40}
I20260928 06:43:33.000567 3684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:32.990922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:40893 (local address 127.18.211.65:33931)
0928 06:43:32.991134 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:32.991144 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:32.991194 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:32.991907 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:32.991915 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:32.992283 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:43:32.992488 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:32.992493 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:32.993626 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:32.993627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:32.994449 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:32.994452 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:32.994989 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:32.994990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:32.995133 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:32.995614 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:43:32.995622 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:43:32.997048 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:43:32.998843 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:43:32.998850 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:43:32.998852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:43:32.999068 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:43:32.999466 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:43:32.999467 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:43:32.999467 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:43:32.999588 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:43:32.999966 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:43:32.999973 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:43:33.000093 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 06:43:33.000172 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 06:43:33.000429 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":129,"thread_start_us":49,"threads_started":1}
W20260928 06:43:33.001528 3296 leader_election.cc:341] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b745a811fae54cd996e42b4fc6f14864 (127.18.211.67:40893): Not found: Tablet not found: 936fc1451c584bad8d4b4ca64390c57a
W20260928 06:43:33.001616 3296 leader_election.cc:341] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b9d2814da9c4fe89992a72a0bbb858c (127.18.211.66:33735): Not found: Tablet not found: 936fc1451c584bad8d4b4ca64390c57a
I20260928 06:43:33.001648 3296 leader_election.cc:302] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dd26eed1e5243c7b65f80d825b02db3; no voters: 1b9d2814da9c4fe89992a72a0bbb858c, b745a811fae54cd996e42b4fc6f14864
I20260928 06:43:33.001763 3683 raft_consensus.cc:2726] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:43:33.683398 3687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:33.683182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.65:48969)
0928 06:43:33.683313 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:33.683352 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":59,"threads_started":1}
I20260928 06:43:33.744266 3688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:33.744067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.66:52689)
0928 06:43:33.744189 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:33.744227 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":57,"threads_started":1}
I20260928 06:43:33.864660 3689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:33.864448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.67:60883)
0928 06:43:33.864581 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:33.864618 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":78,"threads_started":1}
I20260928 06:43:34.597970 3690 raft_consensus.cc:482] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:43:34.598067 3690 raft_consensus.cc:504] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b9d2814da9c4fe89992a72a0bbb858c" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33735 } } peers { permanent_uuid: "b745a811fae54cd996e42b4fc6f14864" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40893 } } peers { permanent_uuid: "5dd26eed1e5243c7b65f80d825b02db3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 40593 } }
I20260928 06:43:34.598201 3690 leader_election.cc:288] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b9d2814da9c4fe89992a72a0bbb858c (127.18.211.66:33735), b745a811fae54cd996e42b4fc6f14864 (127.18.211.67:40893)
W20260928 06:43:34.598624 3296 leader_election.cc:341] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b745a811fae54cd996e42b4fc6f14864 (127.18.211.67:40893): Not found: Tablet not found: 936fc1451c584bad8d4b4ca64390c57a
W20260928 06:43:34.598702 3296 leader_election.cc:341] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b9d2814da9c4fe89992a72a0bbb858c (127.18.211.66:33735): Not found: Tablet not found: 936fc1451c584bad8d4b4ca64390c57a
I20260928 06:43:34.598737 3296 leader_election.cc:302] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dd26eed1e5243c7b65f80d825b02db3; no voters: 1b9d2814da9c4fe89992a72a0bbb858c, b745a811fae54cd996e42b4fc6f14864
I20260928 06:43:34.598825 3690 raft_consensus.cc:2726] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:43:34.684077 3691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:34.683856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.65:53289)
0928 06:43:34.683974 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:34.684012 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260928 06:43:34.744933 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:34.744716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.66:48849)
0928 06:43:34.744859 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:34.744897 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":82,"threads_started":1}
I20260928 06:43:34.865368 3693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:34.865092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.67:35307)
0928 06:43:34.865260 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:34.865298 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":82,"threads_started":1}
I20260928 06:43:35.684682 3694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:35.684462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.65:34303)
0928 06:43:35.684598 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:35.684635 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
I20260928 06:43:35.745563 3695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:35.745359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.66:46777)
0928 06:43:35.745477 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:35.745521 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
I20260928 06:43:35.866065 3696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:35.865846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.67:44111)
0928 06:43:35.865985 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:35.866022 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
I20260928 06:43:36.207947 3697 raft_consensus.cc:482] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:43:36.208063 3697 raft_consensus.cc:504] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b9d2814da9c4fe89992a72a0bbb858c" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33735 } } peers { permanent_uuid: "b745a811fae54cd996e42b4fc6f14864" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40893 } } peers { permanent_uuid: "5dd26eed1e5243c7b65f80d825b02db3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 40593 } }
I20260928 06:43:36.208221 3697 leader_election.cc:288] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b9d2814da9c4fe89992a72a0bbb858c (127.18.211.66:33735), b745a811fae54cd996e42b4fc6f14864 (127.18.211.67:40893)
W20260928 06:43:36.208698 3296 leader_election.cc:341] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b745a811fae54cd996e42b4fc6f14864 (127.18.211.67:40893): Not found: Tablet not found: 936fc1451c584bad8d4b4ca64390c57a
W20260928 06:43:36.208772 3296 leader_election.cc:341] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b9d2814da9c4fe89992a72a0bbb858c (127.18.211.66:33735): Not found: Tablet not found: 936fc1451c584bad8d4b4ca64390c57a
I20260928 06:43:36.208789 3296 leader_election.cc:302] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dd26eed1e5243c7b65f80d825b02db3; no voters: 1b9d2814da9c4fe89992a72a0bbb858c, b745a811fae54cd996e42b4fc6f14864
I20260928 06:43:36.208876 3697 raft_consensus.cc:2726] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:43:36.685348 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:36.685118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.65:50931)
0928 06:43:36.685235 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:36.685272 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
I20260928 06:43:36.746206 3699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:36.746013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.66:43781)
0928 06:43:36.746129 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:36.746166 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
I20260928 06:43:36.866817 3700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:36.866562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.67:46083)
0928 06:43:36.866727 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:36.866772 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":65,"threads_started":1}
I20260928 06:43:37.686012 3701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:37.685790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.65:44045)
0928 06:43:37.685929 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:37.685965 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":84,"threads_started":1}
I20260928 06:43:37.746917 3703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:37.746714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.66:46695)
0928 06:43:37.746832 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:37.746874 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":64,"threads_started":1}
I20260928 06:43:37.867533 3704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:37.867312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.67:45375)
0928 06:43:37.867452 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:37.867491 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":62,"threads_started":1}
I20260928 06:43:38.595628 3705 raft_consensus.cc:482] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:43:38.595733 3705 raft_consensus.cc:504] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b9d2814da9c4fe89992a72a0bbb858c" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33735 } } peers { permanent_uuid: "b745a811fae54cd996e42b4fc6f14864" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40893 } } peers { permanent_uuid: "5dd26eed1e5243c7b65f80d825b02db3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 40593 } }
I20260928 06:43:38.595867 3705 leader_election.cc:288] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b9d2814da9c4fe89992a72a0bbb858c (127.18.211.66:33735), b745a811fae54cd996e42b4fc6f14864 (127.18.211.67:40893)
W20260928 06:43:38.596334 3296 leader_election.cc:341] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b745a811fae54cd996e42b4fc6f14864 (127.18.211.67:40893): Not found: Tablet not found: 936fc1451c584bad8d4b4ca64390c57a
W20260928 06:43:38.596402 3296 leader_election.cc:341] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b9d2814da9c4fe89992a72a0bbb858c (127.18.211.66:33735): Not found: Tablet not found: 936fc1451c584bad8d4b4ca64390c57a
I20260928 06:43:38.596416 3296 leader_election.cc:302] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dd26eed1e5243c7b65f80d825b02db3; no voters: 1b9d2814da9c4fe89992a72a0bbb858c, b745a811fae54cd996e42b4fc6f14864
I20260928 06:43:38.596500 3705 raft_consensus.cc:2726] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:43:38.686806 3706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:38.686532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.65:37369)
0928 06:43:38.686709 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:38.686758 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":88,"threads_started":1}
I20260928 06:43:38.747709 3707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:38.747424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.66:52561)
0928 06:43:38.747580 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:38.747628 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":62,"threads_started":1}
I20260928 06:43:38.868314 3708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:38.868054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.67:40697)
0928 06:43:38.868204 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:38.868252 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":61,"threads_started":1}
I20260928 06:43:39.687533 3709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:39.687333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.65:44343)
0928 06:43:39.687451 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:39.687486 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":57,"threads_started":1}
I20260928 06:43:39.748425 3710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:39.748181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.66:57895)
0928 06:43:39.748317 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:39.748368 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":65,"threads_started":1}
I20260928 06:43:39.869117 3711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:39.868849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.67:50217)
0928 06:43:39.869022 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:39.869069 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":89,"threads_started":1}
I20260928 06:43:40.536549 3712 raft_consensus.cc:482] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:43:40.536669 3712 raft_consensus.cc:504] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b9d2814da9c4fe89992a72a0bbb858c" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33735 } } peers { permanent_uuid: "b745a811fae54cd996e42b4fc6f14864" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40893 } } peers { permanent_uuid: "5dd26eed1e5243c7b65f80d825b02db3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 40593 } }
I20260928 06:43:40.536814 3712 leader_election.cc:288] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b9d2814da9c4fe89992a72a0bbb858c (127.18.211.66:33735), b745a811fae54cd996e42b4fc6f14864 (127.18.211.67:40893)
W20260928 06:43:40.537150 3296 leader_election.cc:341] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b9d2814da9c4fe89992a72a0bbb858c (127.18.211.66:33735): Not found: Tablet not found: 936fc1451c584bad8d4b4ca64390c57a
W20260928 06:43:40.537232 3296 leader_election.cc:341] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b745a811fae54cd996e42b4fc6f14864 (127.18.211.67:40893): Not found: Tablet not found: 936fc1451c584bad8d4b4ca64390c57a
I20260928 06:43:40.537248 3296 leader_election.cc:302] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dd26eed1e5243c7b65f80d825b02db3; no voters: 1b9d2814da9c4fe89992a72a0bbb858c, b745a811fae54cd996e42b4fc6f14864
I20260928 06:43:40.537340 3712 raft_consensus.cc:2726] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:43:40.688311 3713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:40.688054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.65:32831)
0928 06:43:40.688214 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:40.688259 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":63,"threads_started":1}
I20260928 06:43:40.749095 3714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:40.748865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.66:54917)
0928 06:43:40.749011 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:40.749051 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":70,"threads_started":1}
I20260928 06:43:40.869841 3715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:40.869593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.67:36047)
0928 06:43:40.869756 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:40.869796 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":78,"threads_started":1}
I20260928 06:43:41.689106 3716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:41.688842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.65:57255)
0928 06:43:41.689014 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:41.689059 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":92,"threads_started":1}
I20260928 06:43:41.749774 3717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:41.749555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.66:38919)
0928 06:43:41.749689 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:41.749727 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":66,"threads_started":1}
I20260928 06:43:41.870443 3718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:41.870242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.67:46649)
0928 06:43:41.870364 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:41.870398 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
I20260928 06:43:42.689881 3719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:42.689608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.65:47137)
0928 06:43:42.689785 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:42.689832 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":87,"threads_started":1}
I20260928 06:43:42.750494 3720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:42.750192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.66:35605)
0928 06:43:42.750399 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:42.750443 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":80,"threads_started":1}
I20260928 06:43:42.871186 3721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:42.870942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.67:55801)
0928 06:43:42.871097 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:42.871138 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
I20260928 06:43:43.690666 3722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:43.690412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.65:51449)
0928 06:43:43.690570 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:43.690617 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":74,"threads_started":1}
I20260928 06:43:43.751241 3723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:43.750961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.66:39681)
0928 06:43:43.751153 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:43.751195 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":86,"threads_started":1}
I20260928 06:43:43.871906 3724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:43.871659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.67:36391)
0928 06:43:43.871807 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:43.871847 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":86,"threads_started":1}
I20260928 06:43:44.691412 3725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:44.691208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.65:60203)
0928 06:43:44.691327 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:44.691364 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":62,"threads_started":1}
I20260928 06:43:44.751895 3726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:44.751672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.66:48133)
0928 06:43:44.751805 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:44.751848 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":59,"threads_started":1}
I20260928 06:43:44.872592 3727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:44.872350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.67:48261)
0928 06:43:44.872505 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:44.872545 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":79,"threads_started":1}
I20260928 06:43:45.692106 3728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:45.691864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.65:45173)
0928 06:43:45.692011 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:45.692058 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":63,"threads_started":1}
I20260928 06:43:45.752583 3729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:45.752323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.66:55715)
0928 06:43:45.752492 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:45.752535 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":75,"threads_started":1}
I20260928 06:43:45.873325 3730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:45.873042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.67:50671)
0928 06:43:45.873194 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:45.873268 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260928 06:43:46.366281 3731 raft_consensus.cc:482] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:43:46.366381 3731 raft_consensus.cc:504] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b9d2814da9c4fe89992a72a0bbb858c" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33735 } } peers { permanent_uuid: "b745a811fae54cd996e42b4fc6f14864" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40893 } } peers { permanent_uuid: "5dd26eed1e5243c7b65f80d825b02db3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 40593 } }
I20260928 06:43:46.366650 3731 leader_election.cc:288] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b9d2814da9c4fe89992a72a0bbb858c (127.18.211.66:33735), b745a811fae54cd996e42b4fc6f14864 (127.18.211.67:40893)
W20260928 06:43:46.367060 3296 leader_election.cc:341] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1b9d2814da9c4fe89992a72a0bbb858c (127.18.211.66:33735): Not found: Tablet not found: 936fc1451c584bad8d4b4ca64390c57a
W20260928 06:43:46.367156 3296 leader_election.cc:341] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b745a811fae54cd996e42b4fc6f14864 (127.18.211.67:40893): Not found: Tablet not found: 936fc1451c584bad8d4b4ca64390c57a
I20260928 06:43:46.367172 3296 leader_election.cc:302] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5dd26eed1e5243c7b65f80d825b02db3; no voters: 1b9d2814da9c4fe89992a72a0bbb858c, b745a811fae54cd996e42b4fc6f14864
I20260928 06:43:46.367291 3731 raft_consensus.cc:2726] T 936fc1451c584bad8d4b4ca64390c57a P 5dd26eed1e5243c7b65f80d825b02db3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 06:43:46.692811 3732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:46.692614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.65:57957)
0928 06:43:46.692734 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:46.692766 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":65,"threads_started":1}
I20260928 06:43:46.753330 3733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:46.753093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.66:42813)
0928 06:43:46.753249 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:46.753287 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":80,"threads_started":1}
I20260928 06:43:46.868427 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 3241
I20260928 06:43:46.872992 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 3413
I20260928 06:43:46.874459 3734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:46.874160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35839 (local address 127.18.211.67:60251)
0928 06:43:46.874371 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:46.874409 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.126:35839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":51,"threads_started":1}
I20260928 06:43:46.877318 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 3546
2026-09-28T06:43:46Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58187 ms)
[----------] 30 tests from SecurityITest (224360 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:43:46 dist-test-slave-b28r krb5kdc[3737](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:43:46 dist-test-slave-b28r krb5kdc[3737](info): set up 2 sockets
Sep 28 06:43:46 dist-test-slave-b28r krb5kdc[3737](info): commencing operation
krb5kdc: starting...
W20260928 06:43:47.827632 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:43:47 dist-test-slave-b28r krb5kdc[3737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:43:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:43:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.211.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
I20260928 06:43:47.939442 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:39215
--webserver_interface=127.18.211.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:41789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:39215
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:43:48.018800 3750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:43:48.019002 3750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:43:48.019023 3750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:43:48.020228 3750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:43:48.020273 3750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:43:48.020287 3750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:43:48.020298 3750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:43:48.020308 3750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:43:48.021833 3750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:41789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:39215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:39215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:43:48.022054 3750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:43:48.022275 3750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:43:48.026973 3756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:48.026998 3758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:48.027055 3750 server_base.cc:1034] running on GCE node
W20260928 06:43:48.027020 3755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:48.027555 3750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:43:48.027853 3750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:43:48.029011 3750 hybrid_clock.cc:648] HybridClock initialized: now 1790577828028993 us; error 39 us; skew 500 ppm
Sep 28 06:43:48 dist-test-slave-b28r krb5kdc[3737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577828, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:43:48.031455 3750 init.cc:377] Logged in from keytab as kudu/127.18.211.126@KRBTEST.COM (short username kudu)
I20260928 06:43:48.031890 3750 webserver.cc:460] Webserver started at http://127.18.211.126:34577/ using document root <none> and password file <none>
I20260928 06:43:48.032080 3750 fs_manager.cc:359] Metadata directory not provided
I20260928 06:43:48.032119 3750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:43:48.032223 3750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:43:48.033226 3750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "077396bc8fb34c71aa00823551cc5fd3"
format_stamp: "Formatted at 2026-09-28 06:43:48 on dist-test-slave-b28r"
server_key: "5266cb58c2879a4ffb3753ac4e44c92c"
server_key_iv: "b1321b41638b91d253d1dc450e63796b"
server_key_version: "encryptionkey@0"
I20260928 06:43:48.033527 3750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "077396bc8fb34c71aa00823551cc5fd3"
format_stamp: "Formatted at 2026-09-28 06:43:48 on dist-test-slave-b28r"
server_key: "5266cb58c2879a4ffb3753ac4e44c92c"
server_key_iv: "b1321b41638b91d253d1dc450e63796b"
server_key_version: "encryptionkey@0"
I20260928 06:43:48.034726 3750 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:43:48.035480 3765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:48.035686 3750 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:43:48.035776 3750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "077396bc8fb34c71aa00823551cc5fd3"
format_stamp: "Formatted at 2026-09-28 06:43:48 on dist-test-slave-b28r"
server_key: "5266cb58c2879a4ffb3753ac4e44c92c"
server_key_iv: "b1321b41638b91d253d1dc450e63796b"
server_key_version: "encryptionkey@0"
I20260928 06:43:48.035832 3750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:43:48.053216 3750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:43:48.055389 3750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:43:48.055536 3750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:43:48.059265 3750 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:39215
I20260928 06:43:48.059327 3817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:39215 every 8 connection(s)
I20260928 06:43:48.059628 3750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:43:48.060459 3818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:43:48.062727 3818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 077396bc8fb34c71aa00823551cc5fd3: Bootstrap starting.
I20260928 06:43:48.063310 3818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 077396bc8fb34c71aa00823551cc5fd3: Neither blocks nor log segments found. Creating new log.
I20260928 06:43:48.063583 3818 log.cc:824] T 00000000000000000000000000000000 P 077396bc8fb34c71aa00823551cc5fd3: Log is configured to *not* fsync() on all Append() calls
I20260928 06:43:48.063966 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 3750
I20260928 06:43:48.064066 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:43:48.064231 19277 external_mini_cluster.cc:1428] Setting key 784ce172e8adb065d11d7986646ee306
I20260928 06:43:48.064317 3818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 077396bc8fb34c71aa00823551cc5fd3: No bootstrap required, opened a new log
I20260928 06:43:48.066366 3818 raft_consensus.cc:348] T 00000000000000000000000000000000 P 077396bc8fb34c71aa00823551cc5fd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077396bc8fb34c71aa00823551cc5fd3" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 39215 } }
I20260928 06:43:48.066521 3818 raft_consensus.cc:374] T 00000000000000000000000000000000 P 077396bc8fb34c71aa00823551cc5fd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:43:48.066558 3818 raft_consensus.cc:729] T 00000000000000000000000000000000 P 077396bc8fb34c71aa00823551cc5fd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 077396bc8fb34c71aa00823551cc5fd3, State: Initialized, Role: FOLLOWER
I20260928 06:43:48.066684 3818 consensus_queue.cc:260] T 00000000000000000000000000000000 P 077396bc8fb34c71aa00823551cc5fd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077396bc8fb34c71aa00823551cc5fd3" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 39215 } }
I20260928 06:43:48.066766 3818 raft_consensus.cc:388] T 00000000000000000000000000000000 P 077396bc8fb34c71aa00823551cc5fd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:43:48.066807 3818 raft_consensus.cc:482] T 00000000000000000000000000000000 P 077396bc8fb34c71aa00823551cc5fd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:43:48.066855 3818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 077396bc8fb34c71aa00823551cc5fd3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:43:48.067471 3818 raft_consensus.cc:504] T 00000000000000000000000000000000 P 077396bc8fb34c71aa00823551cc5fd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077396bc8fb34c71aa00823551cc5fd3" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 39215 } }
I20260928 06:43:48.067570 3818 leader_election.cc:302] T 00000000000000000000000000000000 P 077396bc8fb34c71aa00823551cc5fd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 077396bc8fb34c71aa00823551cc5fd3; no voters:
I20260928 06:43:48.067735 3818 leader_election.cc:288] T 00000000000000000000000000000000 P 077396bc8fb34c71aa00823551cc5fd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:43:48.067773 3823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 077396bc8fb34c71aa00823551cc5fd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:43:48.067924 3823 raft_consensus.cc:686] T 00000000000000000000000000000000 P 077396bc8fb34c71aa00823551cc5fd3 [term 1 LEADER]: Becoming Leader. State: Replica: 077396bc8fb34c71aa00823551cc5fd3, State: Running, Role: LEADER
I20260928 06:43:48.068001 3818 sys_catalog.cc:564] T 00000000000000000000000000000000 P 077396bc8fb34c71aa00823551cc5fd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:43:48.068125 3823 consensus_queue.cc:237] T 00000000000000000000000000000000 P 077396bc8fb34c71aa00823551cc5fd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077396bc8fb34c71aa00823551cc5fd3" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 39215 } }
I20260928 06:43:48.069227 3825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 077396bc8fb34c71aa00823551cc5fd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "077396bc8fb34c71aa00823551cc5fd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077396bc8fb34c71aa00823551cc5fd3" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 39215 } } }
I20260928 06:43:48.069254 3824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 077396bc8fb34c71aa00823551cc5fd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 077396bc8fb34c71aa00823551cc5fd3. Latest consensus state: current_term: 1 leader_uuid: "077396bc8fb34c71aa00823551cc5fd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077396bc8fb34c71aa00823551cc5fd3" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 39215 } } }
I20260928 06:43:48.069399 3825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 077396bc8fb34c71aa00823551cc5fd3 [sys.catalog]: This master's current role is: LEADER
I20260928 06:43:48.069623 3824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 077396bc8fb34c71aa00823551cc5fd3 [sys.catalog]: This master's current role is: LEADER
W20260928 06:43:48.069913 3838 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 077396bc8fb34c71aa00823551cc5fd3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:43:48.069954 3838 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:43:48.070824 3839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:43:48.071023 3839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 06:43:48 dist-test-slave-b28r krb5kdc[3737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:43:48.072635 3839 catalog_manager.cc:1347] Generated new cluster ID: 15f0341c14a841948a95f0392317f3bc
I20260928 06:43:48.072692 3839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:43:48.075882 3822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:48.064754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51660 (local address 127.18.211.126:39215)
0928 06:43:48.064948 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:48.064956 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:48.064984 (+ 28us) server_negotiation.cc:407] Connection header received
0928 06:43:48.065518 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:48.065525 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:48.065573 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:48.065694 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:48.066308 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.067581 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.068614 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.069201 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.069300 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:48.072040 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:48.072045 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:48.072045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:43:48.072057 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:48.074064 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:48.074418 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:48.074421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:48.074422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:48.074513 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:48.074668 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:48.074669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:48.074669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:48.074945 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:48.075102 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:48.075548 (+ 446us) server_negotiation.cc:299] Negotiation successful
0928 06:43:48.075675 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":90,"threads_started":1}
I20260928 06:43:48.083105 3839 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:43:48.083628 3839 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:43:48.097287 3839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 077396bc8fb34c71aa00823551cc5fd3: Generated new TSK 0
I20260928 06:43:48.097458 3839 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.211.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab.
Entry for principal kudu/127.18.211.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab.
I20260928 06:43:48.125659 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.65:0
--local_ip_for_outbound_sockets=127.18.211.65
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:39215
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:41789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:43:48.224407 3845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:43:48.224742 3845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:43:48.224803 3845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:43:48.226473 3845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:43:48.226521 3845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:43:48.226560 3845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.65
I20260928 06:43:48.228690 3845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:41789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:39215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.65
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:43:48.228945 3845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:43:48.229174 3845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:43:48.234349 3850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:48.234359 3853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:48.234354 3851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:48.234603 3845 server_base.cc:1034] running on GCE node
I20260928 06:43:48.234810 3845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:43:48.235015 3845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:43:48.236157 3845 hybrid_clock.cc:648] HybridClock initialized: now 1790577828236142 us; error 33 us; skew 500 ppm
Sep 28 06:43:48 dist-test-slave-b28r krb5kdc[3737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577828, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:43:48.238584 3845 init.cc:377] Logged in from keytab as kudu/127.18.211.65@KRBTEST.COM (short username kudu)
I20260928 06:43:48.238988 3845 webserver.cc:460] Webserver started at http://127.18.211.65:32935/ using document root <none> and password file <none>
I20260928 06:43:48.239202 3845 fs_manager.cc:359] Metadata directory not provided
I20260928 06:43:48.239248 3845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:43:48.239377 3845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:43:48.240417 3845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-0/data/instance:
uuid: "dbf566031ae74f58997c51cda8755a83"
format_stamp: "Formatted at 2026-09-28 06:43:48 on dist-test-slave-b28r"
server_key: "63be77e9d7f23761a05d745347692ee5"
server_key_iv: "95c612135e5a4cc0bd721305308b687a"
server_key_version: "encryptionkey@0"
I20260928 06:43:48.240737 3845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance:
uuid: "dbf566031ae74f58997c51cda8755a83"
format_stamp: "Formatted at 2026-09-28 06:43:48 on dist-test-slave-b28r"
server_key: "63be77e9d7f23761a05d745347692ee5"
server_key_iv: "95c612135e5a4cc0bd721305308b687a"
server_key_version: "encryptionkey@0"
I20260928 06:43:48.241941 3845 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:43:48.242633 3860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:48.242772 3845 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:43:48.242851 3845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-0/wal
uuid: "dbf566031ae74f58997c51cda8755a83"
format_stamp: "Formatted at 2026-09-28 06:43:48 on dist-test-slave-b28r"
server_key: "63be77e9d7f23761a05d745347692ee5"
server_key_iv: "95c612135e5a4cc0bd721305308b687a"
server_key_version: "encryptionkey@0"
I20260928 06:43:48.242916 3845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:43:48.264304 3845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:43:48.266127 3845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:43:48.266247 3845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:43:48.266487 3845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:43:48.266803 3845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:43:48.266836 3845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:48.266857 3845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:43:48.266881 3845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:48.272467 3845 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.65:34769
I20260928 06:43:48.272526 3973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.65:34769 every 8 connection(s)
I20260928 06:43:48.272854 3845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
Sep 28 06:43:48 dist-test-slave-b28r krb5kdc[3737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577828, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:43:48.281284 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 3845
I20260928 06:43:48.281397 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance
I20260928 06:43:48.281545 3822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:48.273517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:48615 (local address 127.18.211.126:39215)
0928 06:43:48.273575 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:48.273577 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:48.273760 (+ 183us) server_negotiation.cc:407] Connection header received
0928 06:43:48.274250 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:48.274251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:48.274261 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:48.274326 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:48.275268 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.275771 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.276691 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.276913 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.276965 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:48.279021 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:48.279023 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:48.279023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:43:48.279029 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:48.280246 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:48.280608 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:48.280609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:48.280609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:48.280655 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:48.280850 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:48.280851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:48.280851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:48.280946 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:48.280993 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:48.281317 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 06:43:48.281446 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 06:43:48.281576 19277 external_mini_cluster.cc:1428] Setting key 49945dc3fdd81d4b8a775e796d4304cf
I20260928 06:43:48.281647 3976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:48.273487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:39215 (local address 127.18.211.65:48615)
0928 06:43:48.273644 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:48.273688 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:48.274170 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:48.274386 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:48.274389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:48.274944 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:43:48.275192 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:48.275198 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.275841 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.275841 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:48.276621 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:48.276623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.276975 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.276975 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:48.277073 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:48.277485 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:43:48.277493 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:43:48.278929 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:43:48.280317 (+ 1388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:43:48.280324 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:43:48.280325 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:43:48.280549 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:43:48.280698 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:43:48.280699 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:43:48.280699 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:43:48.280808 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:43:48.281023 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:43:48.281029 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:43:48.281153 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 06:43:48.281263 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 06:43:48.281425 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260928 06:43:48.282160 3974 heartbeater.cc:347] Connected to a master server at 127.18.211.126:39215
I20260928 06:43:48.282302 3974 heartbeater.cc:464] Registering TS with master...
I20260928 06:43:48.282475 3974 heartbeater.cc:511] Master 127.18.211.126:39215 requested a full tablet report, sending...
I20260928 06:43:48.283026 3782 ts_manager.cc:194] Registered new tserver with Master: dbf566031ae74f58997c51cda8755a83 (127.18.211.65:34769)
I20260928 06:43:48.283854 3782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.65@KRBTEST.COM'} at 127.18.211.65:48615
WARNING: no policy specified for kudu/127.18.211.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab.
Entry for principal kudu/127.18.211.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab.
I20260928 06:43:48.305420 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.66:0
--local_ip_for_outbound_sockets=127.18.211.66
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:39215
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:41789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:43:48.385423 3979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:43:48.385630 3979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:43:48.385653 3979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:43:48.386801 3979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:43:48.386842 3979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:43:48.386868 3979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.66
I20260928 06:43:48.388383 3979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:41789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:39215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.66
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:43:48.388612 3979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:43:48.388827 3979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:43:48.393630 3984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:48.393718 3979 server_base.cc:1034] running on GCE node
W20260928 06:43:48.393625 3987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:48.393625 3985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:48.393996 3979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:43:48.394254 3979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:43:48.395414 3979 hybrid_clock.cc:648] HybridClock initialized: now 1790577828395405 us; error 30 us; skew 500 ppm
Sep 28 06:43:48 dist-test-slave-b28r krb5kdc[3737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577828, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:43:48.397748 3979 init.cc:377] Logged in from keytab as kudu/127.18.211.66@KRBTEST.COM (short username kudu)
I20260928 06:43:48.398137 3979 webserver.cc:460] Webserver started at http://127.18.211.66:37875/ using document root <none> and password file <none>
I20260928 06:43:48.398377 3979 fs_manager.cc:359] Metadata directory not provided
I20260928 06:43:48.398424 3979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:43:48.398543 3979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:43:48.399605 3979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-1/data/instance:
uuid: "17ba4c282317409e887de72b420833c5"
format_stamp: "Formatted at 2026-09-28 06:43:48 on dist-test-slave-b28r"
server_key: "de1b89edf01c0c04ba5aeaa379bbbff1"
server_key_iv: "a51ccde124ae8872365e0087efa7c03b"
server_key_version: "encryptionkey@0"
I20260928 06:43:48.399924 3979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance:
uuid: "17ba4c282317409e887de72b420833c5"
format_stamp: "Formatted at 2026-09-28 06:43:48 on dist-test-slave-b28r"
server_key: "de1b89edf01c0c04ba5aeaa379bbbff1"
server_key_iv: "a51ccde124ae8872365e0087efa7c03b"
server_key_version: "encryptionkey@0"
I20260928 06:43:48.401228 3979 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:43:48.401880 3994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:48.402029 3979 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:43:48.402105 3979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-1/wal
uuid: "17ba4c282317409e887de72b420833c5"
format_stamp: "Formatted at 2026-09-28 06:43:48 on dist-test-slave-b28r"
server_key: "de1b89edf01c0c04ba5aeaa379bbbff1"
server_key_iv: "a51ccde124ae8872365e0087efa7c03b"
server_key_version: "encryptionkey@0"
I20260928 06:43:48.402204 3979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:43:48.416962 3979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:43:48.419294 3979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:43:48.419435 3979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:43:48.419647 3979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:43:48.419934 3979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:43:48.419967 3979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:48.419988 3979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:43:48.420010 3979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:48.425499 3979 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.66:46233
I20260928 06:43:48.425560 4107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.66:46233 every 8 connection(s)
I20260928 06:43:48.425848 3979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
I20260928 06:43:48.430487 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 3979
I20260928 06:43:48.430604 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance
I20260928 06:43:48.430841 19277 external_mini_cluster.cc:1428] Setting key f431a3c7da36262e9070c089539195db
Sep 28 06:43:48 dist-test-slave-b28r krb5kdc[3737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577828, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:43:48.435168 3822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:48.426520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:43943 (local address 127.18.211.126:39215)
0928 06:43:48.426583 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:48.426585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:48.426772 (+ 187us) server_negotiation.cc:407] Connection header received
0928 06:43:48.427332 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:48.427333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:48.427346 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:48.427398 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:48.428317 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.428814 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.429781 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.429989 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.430045 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:48.432621 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:48.432623 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:48.432623 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:43:48.432630 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:48.433898 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:48.434265 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:48.434266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:48.434266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:48.434330 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:48.434521 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:48.434522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:48.434522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:48.434650 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:48.434707 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:48.434952 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 06:43:48.435046 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 06:43:48.435254 4110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:48.426491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:39215 (local address 127.18.211.66:43943)
0928 06:43:48.426656 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:48.426691 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:48.427196 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:48.427469 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:48.427473 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:48.428030 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:43:48.428220 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:48.428224 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.428896 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.428897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:48.429696 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:48.429698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.430057 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.430058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:48.430182 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:48.430826 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:43:48.430836 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:43:48.432521 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:43:48.433976 (+ 1455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:43:48.433982 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:43:48.433985 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:43:48.434201 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:43:48.434384 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:43:48.434385 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:43:48.434385 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:43:48.434475 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:43:48.434704 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:43:48.434708 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:43:48.434836 (+ 128us) client_negotiation.cc:769] Sending connection context
0928 06:43:48.434912 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 06:43:48.435052 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260928 06:43:48.435719 4108 heartbeater.cc:347] Connected to a master server at 127.18.211.126:39215
I20260928 06:43:48.435827 4108 heartbeater.cc:464] Registering TS with master...
I20260928 06:43:48.435993 4108 heartbeater.cc:511] Master 127.18.211.126:39215 requested a full tablet report, sending...
I20260928 06:43:48.436345 3782 ts_manager.cc:194] Registered new tserver with Master: 17ba4c282317409e887de72b420833c5 (127.18.211.66:46233)
I20260928 06:43:48.436842 3782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.66@KRBTEST.COM'} at 127.18.211.66:43943
WARNING: no policy specified for kudu/127.18.211.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab.
Entry for principal kudu/127.18.211.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab.
I20260928 06:43:48.455196 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.67:0
--local_ip_for_outbound_sockets=127.18.211.67
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:39215
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:41789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:43:48.534839 4113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:43:48.535061 4113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:43:48.535118 4113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:43:48.536317 4113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:43:48.536366 4113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:43:48.536402 4113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.67
I20260928 06:43:48.537926 4113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:41789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:39215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.67
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:43:48.538173 4113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:43:48.538400 4113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:43:48.543256 4121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:48.543331 4118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:48.543298 4113 server_base.cc:1034] running on GCE node
W20260928 06:43:48.543265 4119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:48.543627 4113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:43:48.543841 4113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:43:48.544996 4113 hybrid_clock.cc:648] HybridClock initialized: now 1790577828544980 us; error 28 us; skew 500 ppm
Sep 28 06:43:48 dist-test-slave-b28r krb5kdc[3737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577828, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:43:48.547691 4113 init.cc:377] Logged in from keytab as kudu/127.18.211.67@KRBTEST.COM (short username kudu)
I20260928 06:43:48.548135 4113 webserver.cc:460] Webserver started at http://127.18.211.67:39455/ using document root <none> and password file <none>
I20260928 06:43:48.548398 4113 fs_manager.cc:359] Metadata directory not provided
I20260928 06:43:48.548447 4113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:43:48.548573 4113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:43:48.549611 4113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-2/data/instance:
uuid: "e8d96ce4e2fe40f6b88d42ad880ea90f"
format_stamp: "Formatted at 2026-09-28 06:43:48 on dist-test-slave-b28r"
server_key: "8316529b1b6a3901a487459468c1b641"
server_key_iv: "66861b3e8d912615dc29396a15b2e307"
server_key_version: "encryptionkey@0"
I20260928 06:43:48.549933 4113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance:
uuid: "e8d96ce4e2fe40f6b88d42ad880ea90f"
format_stamp: "Formatted at 2026-09-28 06:43:48 on dist-test-slave-b28r"
server_key: "8316529b1b6a3901a487459468c1b641"
server_key_iv: "66861b3e8d912615dc29396a15b2e307"
server_key_version: "encryptionkey@0"
I20260928 06:43:48.551173 4113 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:43:48.551975 4128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:48.552134 4113 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:43:48.552215 4113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-2/wal
uuid: "e8d96ce4e2fe40f6b88d42ad880ea90f"
format_stamp: "Formatted at 2026-09-28 06:43:48 on dist-test-slave-b28r"
server_key: "8316529b1b6a3901a487459468c1b641"
server_key_iv: "66861b3e8d912615dc29396a15b2e307"
server_key_version: "encryptionkey@0"
I20260928 06:43:48.552320 4113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:43:48.561460 4113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:43:48.563401 4113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:43:48.563531 4113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:43:48.563735 4113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:43:48.564019 4113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:43:48.564069 4113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:48.564103 4113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:43:48.564127 4113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:48.569686 4113 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.67:40369
I20260928 06:43:48.569778 4244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.67:40369 every 8 connection(s)
I20260928 06:43:48.570050 4113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:43:48 dist-test-slave-b28r krb5kdc[3737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577828, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:43:48.578945 3822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:48.570684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:43935 (local address 127.18.211.126:39215)
0928 06:43:48.570731 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:48.570732 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:48.570941 (+ 209us) server_negotiation.cc:407] Connection header received
0928 06:43:48.571456 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:48.571457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:48.571465 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:48.571520 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:48.572388 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.572882 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.573918 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.574122 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.574184 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:48.576314 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:48.576316 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:48.576316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:43:48.576324 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:48.577600 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:48.578060 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:48.578061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:48.578062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:48.578127 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:48.578391 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:48.578393 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:48.578393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:48.578528 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:48.578588 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:48.578813 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 06:43:48.578880 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:43:48.579125 4247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:48.570667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:39215 (local address 127.18.211.67:43935)
0928 06:43:48.570833 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:48.570875 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:48.571375 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:48.571580 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:48.571583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:48.572131 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:43:48.572322 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:48.572327 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.572946 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.572947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:48.573823 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:48.573826 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.574190 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.574191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:48.574284 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:48.574726 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:43:48.574734 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:43:48.576223 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:43:48.577674 (+ 1451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:43:48.577683 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:43:48.577685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:43:48.577982 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:43:48.578187 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:43:48.578188 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:43:48.578188 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:43:48.578309 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:43:48.578587 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:43:48.578591 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:43:48.578703 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 06:43:48.578799 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 06:43:48.578915 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":70,"threads_started":1}
I20260928 06:43:48.579563 4245 heartbeater.cc:347] Connected to a master server at 127.18.211.126:39215
I20260928 06:43:48.579674 4245 heartbeater.cc:464] Registering TS with master...
I20260928 06:43:48.579793 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 4113
I20260928 06:43:48.579859 4245 heartbeater.cc:511] Master 127.18.211.126:39215 requested a full tablet report, sending...
I20260928 06:43:48.579900 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance
I20260928 06:43:48.580071 19277 external_mini_cluster.cc:1428] Setting key a93c78b13140132b8ead6fbe42eb9c6b
I20260928 06:43:48.580219 3782 ts_manager.cc:194] Registered new tserver with Master: e8d96ce4e2fe40f6b88d42ad880ea90f (127.18.211.67:40369)
I20260928 06:43:48.580631 3782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.67@KRBTEST.COM'} at 127.18.211.67:43935
I20260928 06:43:48.580840 19277 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:43:48.587430 3822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:48.582078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51666 (local address 127.18.211.126:39215)
0928 06:43:48.582115 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:48.582117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:48.582215 (+ 98us) server_negotiation.cc:407] Connection header received
0928 06:43:48.582284 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:48.582285 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:48.582295 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:48.582346 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:48.582853 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.583368 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.584064 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.584249 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.584308 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:48.585356 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:48.585358 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:48.585358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:43:48.585365 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:48.586363 (+ 998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:48.586652 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:48.586653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:48.586653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:48.586704 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:48.586878 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:48.586879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:48.586879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:48.587021 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:48.587139 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:48.587264 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 06:43:48.587344 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:43:48.588964 3782 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:43:48.590641 3782 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:43:48.596787 4254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:48.592623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:40369 (local address 127.0.0.1:50806)
0928 06:43:48.592758 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:48.592801 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:48.592868 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:48.593269 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:48.593272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:48.593274 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:43:48.593349 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:48.593350 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.594346 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.594347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:48.595714 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:48.595717 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.596519 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.596520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:48.596605 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:48.596619 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 06:43:48.596661 (+ 42us) client_negotiation.cc:240] Negotiation successful
0928 06:43:48.596709 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":39,"thread_start_us":63,"threads_started":1}
I20260928 06:43:48.597288 4256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:48.592947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:34769 (local address 127.0.0.1:57620)
0928 06:43:48.593350 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:48.593360 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:48.593403 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:48.593608 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:48.593609 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:48.593611 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:43:48.593723 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:48.593725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.594656 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.594658 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:48.596122 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:48.596125 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.597075 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.597076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:48.597143 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:48.597148 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:43:48.597191 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 06:43:48.597227 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":40,"threads_started":1}
I20260928 06:43:48.597301 4255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:48.592714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50806 (local address 127.18.211.67:40369)
0928 06:43:48.593082 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:48.593088 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:48.593099 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:43:48.593139 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:48.593141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:48.593157 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:48.593277 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:43:48.593425 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.594253 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.595812 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.596425 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.597074 (+ 649us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:48.597134 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:48.597159 (+ 25us) server_negotiation.cc:299] Negotiation successful
0928 06:43:48.597199 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":111,"threads_started":1}
I20260928 06:43:48.597378 4257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:48.592815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:46233 (local address 127.0.0.1:48706)
0928 06:43:48.593017 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:48.593026 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:48.593061 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:48.593267 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:48.593270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:48.593273 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:43:48.593347 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:48.593349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.595500 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.595501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:48.596447 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:48.596449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.597139 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.597140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:48.597211 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:48.597216 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:43:48.597265 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 06:43:48.597306 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":4,"thread_start_us":49,"threads_started":1}
I20260928 06:43:48.597398 4258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:48.592933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48706 (local address 127.18.211.66:46233)
0928 06:43:48.593116 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:48.593119 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:48.593128 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:43:48.593161 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:48.593163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:48.593175 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:48.593253 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:43:48.594352 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.595393 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.596522 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.597060 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.597128 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:48.597186 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:48.597283 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 06:43:48.597323 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":80,"threads_started":1}
I20260928 06:43:48.597579 4259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:48.593079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57620 (local address 127.18.211.65:34769)
0928 06:43:48.593467 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:48.593471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:48.593481 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:43:48.593516 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:48.593518 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:48.593533 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:48.593599 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:43:48.593792 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.594554 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.596206 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.596912 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.596978 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:48.597042 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:48.597445 (+ 403us) server_negotiation.cc:299] Negotiation successful
0928 06:43:48.597501 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":92,"threads_started":1}
I20260928 06:43:48.598133 4042 tablet_service.cc:1467] Processing CreateTablet for tablet 9a62f245f09345b499afc5f8044ead6b (DEFAULT_TABLE table=test-table [id=586027b91786425480e0320d28756a24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:43:48.598155 3908 tablet_service.cc:1467] Processing CreateTablet for tablet 9a62f245f09345b499afc5f8044ead6b (DEFAULT_TABLE table=test-table [id=586027b91786425480e0320d28756a24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:43:48.598429 3908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a62f245f09345b499afc5f8044ead6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:43:48.598433 4042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a62f245f09345b499afc5f8044ead6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:43:48.601135 4260 tablet_bootstrap.cc:492] T 9a62f245f09345b499afc5f8044ead6b P 17ba4c282317409e887de72b420833c5: Bootstrap starting.
I20260928 06:43:48.601047 4179 tablet_service.cc:1467] Processing CreateTablet for tablet 9a62f245f09345b499afc5f8044ead6b (DEFAULT_TABLE table=test-table [id=586027b91786425480e0320d28756a24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:43:48.601361 4179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a62f245f09345b499afc5f8044ead6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:43:48.601761 4260 tablet_bootstrap.cc:654] T 9a62f245f09345b499afc5f8044ead6b P 17ba4c282317409e887de72b420833c5: Neither blocks nor log segments found. Creating new log.
I20260928 06:43:48.602293 4260 log.cc:824] T 9a62f245f09345b499afc5f8044ead6b P 17ba4c282317409e887de72b420833c5: Log is configured to *not* fsync() on all Append() calls
I20260928 06:43:48.602646 4261 tablet_bootstrap.cc:492] T 9a62f245f09345b499afc5f8044ead6b P dbf566031ae74f58997c51cda8755a83: Bootstrap starting.
I20260928 06:43:48.602981 4260 tablet_bootstrap.cc:492] T 9a62f245f09345b499afc5f8044ead6b P 17ba4c282317409e887de72b420833c5: No bootstrap required, opened a new log
I20260928 06:43:48.603024 4260 ts_tablet_manager.cc:1397] T 9a62f245f09345b499afc5f8044ead6b P 17ba4c282317409e887de72b420833c5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:43:48.603317 4261 tablet_bootstrap.cc:654] T 9a62f245f09345b499afc5f8044ead6b P dbf566031ae74f58997c51cda8755a83: Neither blocks nor log segments found. Creating new log.
I20260928 06:43:48.603871 4261 log.cc:824] T 9a62f245f09345b499afc5f8044ead6b P dbf566031ae74f58997c51cda8755a83: Log is configured to *not* fsync() on all Append() calls
I20260928 06:43:48.604521 4264 tablet_bootstrap.cc:492] T 9a62f245f09345b499afc5f8044ead6b P e8d96ce4e2fe40f6b88d42ad880ea90f: Bootstrap starting.
I20260928 06:43:48.604696 4261 tablet_bootstrap.cc:492] T 9a62f245f09345b499afc5f8044ead6b P dbf566031ae74f58997c51cda8755a83: No bootstrap required, opened a new log
I20260928 06:43:48.604765 4261 ts_tablet_manager.cc:1397] T 9a62f245f09345b499afc5f8044ead6b P dbf566031ae74f58997c51cda8755a83: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:43:48.604755 4260 raft_consensus.cc:348] T 9a62f245f09345b499afc5f8044ead6b P 17ba4c282317409e887de72b420833c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d96ce4e2fe40f6b88d42ad880ea90f" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40369 } } peers { permanent_uuid: "17ba4c282317409e887de72b420833c5" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46233 } } peers { permanent_uuid: "dbf566031ae74f58997c51cda8755a83" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 34769 } }
I20260928 06:43:48.604934 4260 raft_consensus.cc:374] T 9a62f245f09345b499afc5f8044ead6b P 17ba4c282317409e887de72b420833c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:43:48.604969 4260 raft_consensus.cc:729] T 9a62f245f09345b499afc5f8044ead6b P 17ba4c282317409e887de72b420833c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17ba4c282317409e887de72b420833c5, State: Initialized, Role: FOLLOWER
I20260928 06:43:48.605103 4260 consensus_queue.cc:260] T 9a62f245f09345b499afc5f8044ead6b P 17ba4c282317409e887de72b420833c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d96ce4e2fe40f6b88d42ad880ea90f" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40369 } } peers { permanent_uuid: "17ba4c282317409e887de72b420833c5" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46233 } } peers { permanent_uuid: "dbf566031ae74f58997c51cda8755a83" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 34769 } }
I20260928 06:43:48.605188 4264 tablet_bootstrap.cc:654] T 9a62f245f09345b499afc5f8044ead6b P e8d96ce4e2fe40f6b88d42ad880ea90f: Neither blocks nor log segments found. Creating new log.
I20260928 06:43:48.605388 4260 ts_tablet_manager.cc:1428] T 9a62f245f09345b499afc5f8044ead6b P 17ba4c282317409e887de72b420833c5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:43:48.605454 4108 heartbeater.cc:503] Master 127.18.211.126:39215 was elected leader, sending a full tablet report...
I20260928 06:43:48.605711 4264 log.cc:824] T 9a62f245f09345b499afc5f8044ead6b P e8d96ce4e2fe40f6b88d42ad880ea90f: Log is configured to *not* fsync() on all Append() calls
I20260928 06:43:48.606531 4264 tablet_bootstrap.cc:492] T 9a62f245f09345b499afc5f8044ead6b P e8d96ce4e2fe40f6b88d42ad880ea90f: No bootstrap required, opened a new log
I20260928 06:43:48.606618 4264 ts_tablet_manager.cc:1397] T 9a62f245f09345b499afc5f8044ead6b P e8d96ce4e2fe40f6b88d42ad880ea90f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:43:48.606647 4261 raft_consensus.cc:348] T 9a62f245f09345b499afc5f8044ead6b P dbf566031ae74f58997c51cda8755a83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d96ce4e2fe40f6b88d42ad880ea90f" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40369 } } peers { permanent_uuid: "17ba4c282317409e887de72b420833c5" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46233 } } peers { permanent_uuid: "dbf566031ae74f58997c51cda8755a83" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 34769 } }
I20260928 06:43:48.606844 4261 raft_consensus.cc:374] T 9a62f245f09345b499afc5f8044ead6b P dbf566031ae74f58997c51cda8755a83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:43:48.606889 4261 raft_consensus.cc:729] T 9a62f245f09345b499afc5f8044ead6b P dbf566031ae74f58997c51cda8755a83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbf566031ae74f58997c51cda8755a83, State: Initialized, Role: FOLLOWER
I20260928 06:43:48.606998 4261 consensus_queue.cc:260] T 9a62f245f09345b499afc5f8044ead6b P dbf566031ae74f58997c51cda8755a83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d96ce4e2fe40f6b88d42ad880ea90f" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40369 } } peers { permanent_uuid: "17ba4c282317409e887de72b420833c5" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46233 } } peers { permanent_uuid: "dbf566031ae74f58997c51cda8755a83" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 34769 } }
I20260928 06:43:48.607254 4261 ts_tablet_manager.cc:1428] T 9a62f245f09345b499afc5f8044ead6b P dbf566031ae74f58997c51cda8755a83: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260928 06:43:48.607287 3974 heartbeater.cc:503] Master 127.18.211.126:39215 was elected leader, sending a full tablet report...
I20260928 06:43:48.608412 4264 raft_consensus.cc:348] T 9a62f245f09345b499afc5f8044ead6b P e8d96ce4e2fe40f6b88d42ad880ea90f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d96ce4e2fe40f6b88d42ad880ea90f" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40369 } } peers { permanent_uuid: "17ba4c282317409e887de72b420833c5" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46233 } } peers { permanent_uuid: "dbf566031ae74f58997c51cda8755a83" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 34769 } }
I20260928 06:43:48.608541 4264 raft_consensus.cc:374] T 9a62f245f09345b499afc5f8044ead6b P e8d96ce4e2fe40f6b88d42ad880ea90f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:43:48.608572 4264 raft_consensus.cc:729] T 9a62f245f09345b499afc5f8044ead6b P e8d96ce4e2fe40f6b88d42ad880ea90f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d96ce4e2fe40f6b88d42ad880ea90f, State: Initialized, Role: FOLLOWER
I20260928 06:43:48.608675 4264 consensus_queue.cc:260] T 9a62f245f09345b499afc5f8044ead6b P e8d96ce4e2fe40f6b88d42ad880ea90f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d96ce4e2fe40f6b88d42ad880ea90f" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40369 } } peers { permanent_uuid: "17ba4c282317409e887de72b420833c5" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46233 } } peers { permanent_uuid: "dbf566031ae74f58997c51cda8755a83" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 34769 } }
I20260928 06:43:48.608894 4264 ts_tablet_manager.cc:1428] T 9a62f245f09345b499afc5f8044ead6b P e8d96ce4e2fe40f6b88d42ad880ea90f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 06:43:48.608952 4245 heartbeater.cc:503] Master 127.18.211.126:39215 was elected leader, sending a full tablet report...
W20260928 06:43:48.676442 4109 tablet.cc:2369] T 9a62f245f09345b499afc5f8044ead6b P 17ba4c282317409e887de72b420833c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:43:48.773687 3975 tablet.cc:2369] T 9a62f245f09345b499afc5f8044ead6b P dbf566031ae74f58997c51cda8755a83: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:43:48.820632 4246 tablet.cc:2369] T 9a62f245f09345b499afc5f8044ead6b P e8d96ce4e2fe40f6b88d42ad880ea90f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:43:48.883589 4268 raft_consensus.cc:482] T 9a62f245f09345b499afc5f8044ead6b P e8d96ce4e2fe40f6b88d42ad880ea90f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:43:48.883723 4268 raft_consensus.cc:504] T 9a62f245f09345b499afc5f8044ead6b P e8d96ce4e2fe40f6b88d42ad880ea90f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d96ce4e2fe40f6b88d42ad880ea90f" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40369 } } peers { permanent_uuid: "17ba4c282317409e887de72b420833c5" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46233 } } peers { permanent_uuid: "dbf566031ae74f58997c51cda8755a83" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 34769 } }
I20260928 06:43:48.884196 4268 leader_election.cc:288] T 9a62f245f09345b499afc5f8044ead6b P e8d96ce4e2fe40f6b88d42ad880ea90f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17ba4c282317409e887de72b420833c5 (127.18.211.66:46233), dbf566031ae74f58997c51cda8755a83 (127.18.211.65:34769)
I20260928 06:43:48.887171 4247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:48.884394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:46233 (local address 127.18.211.67:37117)
0928 06:43:48.884446 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:48.884458 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:48.884505 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:48.884671 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:48.884672 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:48.884674 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:43:48.884749 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:48.884751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.885511 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.885512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:48.886302 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:48.886304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.886907 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.886908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:48.886987 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:48.886991 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:43:48.887041 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 06:43:48.887104 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 06:43:48.887172 4258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:48.884460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:37117 (local address 127.18.211.66:46233)
0928 06:43:48.884514 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:48.884516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:48.884528 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:43:48.884556 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:48.884557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:48.884566 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:48.884649 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:43:48.884848 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.885431 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.886397 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.886837 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.886890 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:48.886897 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:48.887049 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 06:43:48.887104 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 06:43:48.887308 4269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:48.884514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:34769 (local address 127.18.211.67:44743)
0928 06:43:48.884759 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:48.884767 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:48.884839 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:48.884964 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:48.884965 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:48.884966 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:43:48.885032 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:48.885034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.885720 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.885721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:48.886511 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:48.886514 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.887055 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.887056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:48.887146 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:48.887150 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:43:48.887204 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 06:43:48.887239 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":100,"threads_started":1}
I20260928 06:43:48.887331 4259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:48.884672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:44743 (local address 127.18.211.65:34769)
0928 06:43:48.884736 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:48.884738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:48.884821 (+ 83us) server_negotiation.cc:407] Connection header received
0928 06:43:48.884896 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:48.884898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:48.884905 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:48.884945 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:43:48.885090 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.885645 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.886592 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.886973 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.887042 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:48.887050 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:48.887216 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 06:43:48.887256 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 06:43:48.887669 3928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a62f245f09345b499afc5f8044ead6b" candidate_uuid: "e8d96ce4e2fe40f6b88d42ad880ea90f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbf566031ae74f58997c51cda8755a83" is_pre_election: true
I20260928 06:43:48.887732 4062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a62f245f09345b499afc5f8044ead6b" candidate_uuid: "e8d96ce4e2fe40f6b88d42ad880ea90f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17ba4c282317409e887de72b420833c5" is_pre_election: true
I20260928 06:43:48.887809 3928 raft_consensus.cc:2454] T 9a62f245f09345b499afc5f8044ead6b P dbf566031ae74f58997c51cda8755a83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8d96ce4e2fe40f6b88d42ad880ea90f in term 0.
I20260928 06:43:48.887871 4062 raft_consensus.cc:2454] T 9a62f245f09345b499afc5f8044ead6b P 17ba4c282317409e887de72b420833c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8d96ce4e2fe40f6b88d42ad880ea90f in term 0.
I20260928 06:43:48.888062 4129 leader_election.cc:302] T 9a62f245f09345b499afc5f8044ead6b P e8d96ce4e2fe40f6b88d42ad880ea90f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dbf566031ae74f58997c51cda8755a83, e8d96ce4e2fe40f6b88d42ad880ea90f; no voters:
I20260928 06:43:48.888185 4268 raft_consensus.cc:2781] T 9a62f245f09345b499afc5f8044ead6b P e8d96ce4e2fe40f6b88d42ad880ea90f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:43:48.888245 4268 raft_consensus.cc:482] T 9a62f245f09345b499afc5f8044ead6b P e8d96ce4e2fe40f6b88d42ad880ea90f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:43:48.888268 4268 raft_consensus.cc:3037] T 9a62f245f09345b499afc5f8044ead6b P e8d96ce4e2fe40f6b88d42ad880ea90f [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:43:48.888996 4268 raft_consensus.cc:504] T 9a62f245f09345b499afc5f8044ead6b P e8d96ce4e2fe40f6b88d42ad880ea90f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d96ce4e2fe40f6b88d42ad880ea90f" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40369 } } peers { permanent_uuid: "17ba4c282317409e887de72b420833c5" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46233 } } peers { permanent_uuid: "dbf566031ae74f58997c51cda8755a83" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 34769 } }
I20260928 06:43:48.889125 4268 leader_election.cc:288] T 9a62f245f09345b499afc5f8044ead6b P e8d96ce4e2fe40f6b88d42ad880ea90f [CANDIDATE]: Term 1 election: Requested vote from peers 17ba4c282317409e887de72b420833c5 (127.18.211.66:46233), dbf566031ae74f58997c51cda8755a83 (127.18.211.65:34769)
I20260928 06:43:48.889266 4062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a62f245f09345b499afc5f8044ead6b" candidate_uuid: "e8d96ce4e2fe40f6b88d42ad880ea90f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17ba4c282317409e887de72b420833c5"
I20260928 06:43:48.889334 4062 raft_consensus.cc:3037] T 9a62f245f09345b499afc5f8044ead6b P 17ba4c282317409e887de72b420833c5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:43:48.889348 3928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a62f245f09345b499afc5f8044ead6b" candidate_uuid: "e8d96ce4e2fe40f6b88d42ad880ea90f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbf566031ae74f58997c51cda8755a83"
I20260928 06:43:48.889436 3928 raft_consensus.cc:3037] T 9a62f245f09345b499afc5f8044ead6b P dbf566031ae74f58997c51cda8755a83 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:43:48.890044 4062 raft_consensus.cc:2454] T 9a62f245f09345b499afc5f8044ead6b P 17ba4c282317409e887de72b420833c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8d96ce4e2fe40f6b88d42ad880ea90f in term 1.
I20260928 06:43:48.890177 3928 raft_consensus.cc:2454] T 9a62f245f09345b499afc5f8044ead6b P dbf566031ae74f58997c51cda8755a83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8d96ce4e2fe40f6b88d42ad880ea90f in term 1.
I20260928 06:43:48.890201 4129 leader_election.cc:302] T 9a62f245f09345b499afc5f8044ead6b P e8d96ce4e2fe40f6b88d42ad880ea90f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17ba4c282317409e887de72b420833c5, e8d96ce4e2fe40f6b88d42ad880ea90f; no voters:
I20260928 06:43:48.890278 4268 raft_consensus.cc:2781] T 9a62f245f09345b499afc5f8044ead6b P e8d96ce4e2fe40f6b88d42ad880ea90f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:43:48.890415 4268 raft_consensus.cc:686] T 9a62f245f09345b499afc5f8044ead6b P e8d96ce4e2fe40f6b88d42ad880ea90f [term 1 LEADER]: Becoming Leader. State: Replica: e8d96ce4e2fe40f6b88d42ad880ea90f, State: Running, Role: LEADER
I20260928 06:43:48.890530 4268 consensus_queue.cc:237] T 9a62f245f09345b499afc5f8044ead6b P e8d96ce4e2fe40f6b88d42ad880ea90f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d96ce4e2fe40f6b88d42ad880ea90f" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40369 } } peers { permanent_uuid: "17ba4c282317409e887de72b420833c5" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46233 } } peers { permanent_uuid: "dbf566031ae74f58997c51cda8755a83" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 34769 } }
I20260928 06:43:48.891320 3782 catalog_manager.cc:5495] T 9a62f245f09345b499afc5f8044ead6b P e8d96ce4e2fe40f6b88d42ad880ea90f reported cstate change: term changed from 0 to 1, leader changed from <none> to e8d96ce4e2fe40f6b88d42ad880ea90f (127.18.211.67). New cstate: current_term: 1 leader_uuid: "e8d96ce4e2fe40f6b88d42ad880ea90f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8d96ce4e2fe40f6b88d42ad880ea90f" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17ba4c282317409e887de72b420833c5" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbf566031ae74f58997c51cda8755a83" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 34769 } health_report { overall_health: UNKNOWN } } }
I20260928 06:43:48.948045 4255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:48.945779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44306 (local address 127.18.211.67:40369)
0928 06:43:48.945823 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:48.945826 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:48.945847 (+ 21us) server_negotiation.cc:407] Connection header received
0928 06:43:48.945894 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:48.945895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:48.945920 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:48.945976 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:43:48.946084 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.946589 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.947340 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:48.947537 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:48.947588 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:48.947697 (+ 109us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:43:48.947821 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:43:48.947890 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:48.947924 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 06:43:48.947943 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:43:48.949827 4062 raft_consensus.cc:1261] T 9a62f245f09345b499afc5f8044ead6b P 17ba4c282317409e887de72b420833c5 [term 1 FOLLOWER]: Refusing update from remote peer e8d96ce4e2fe40f6b88d42ad880ea90f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:43:48.949834 3928 raft_consensus.cc:1261] T 9a62f245f09345b499afc5f8044ead6b P dbf566031ae74f58997c51cda8755a83 [term 1 FOLLOWER]: Refusing update from remote peer e8d96ce4e2fe40f6b88d42ad880ea90f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:43:48.950179 4270 consensus_queue.cc:1035] T 9a62f245f09345b499afc5f8044ead6b P e8d96ce4e2fe40f6b88d42ad880ea90f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17ba4c282317409e887de72b420833c5" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 46233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:43:48.950314 4268 consensus_queue.cc:1035] T 9a62f245f09345b499afc5f8044ead6b P e8d96ce4e2fe40f6b88d42ad880ea90f [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbf566031ae74f58997c51cda8755a83" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 34769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:43:48.952749 4274 mvcc.cc:205] Tried to move back new op lower bound from 7334206787375702016 to 7334206787136135168. Current Snapshot: MvccSnapshot[applied={T|T < 7334206787375702016}]
I20260928 06:43:48.954635 3781 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:43:48.954728 3781 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:43:48.955502 3781 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 077396bc8fb34c71aa00823551cc5fd3: Sending DeleteTablet for 3 replicas of tablet 9a62f245f09345b499afc5f8044ead6b
I20260928 06:43:48.955803 4179 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a62f245f09345b499afc5f8044ead6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:43:48 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:50806
I20260928 06:43:48.955840 4042 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a62f245f09345b499afc5f8044ead6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:43:48 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:48706
I20260928 06:43:48.956080 4283 tablet_replica.cc:330] T 9a62f245f09345b499afc5f8044ead6b P e8d96ce4e2fe40f6b88d42ad880ea90f: stopping tablet replica
I20260928 06:43:48.956076 3908 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a62f245f09345b499afc5f8044ead6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:43:48 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:57620
I20260928 06:43:48.956235 4283 raft_consensus.cc:2229] T 9a62f245f09345b499afc5f8044ead6b P e8d96ce4e2fe40f6b88d42ad880ea90f [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:43:48.956377 4283 raft_consensus.cc:2258] T 9a62f245f09345b499afc5f8044ead6b P e8d96ce4e2fe40f6b88d42ad880ea90f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:43:48.956372 4285 tablet_replica.cc:330] T 9a62f245f09345b499afc5f8044ead6b P dbf566031ae74f58997c51cda8755a83: stopping tablet replica
I20260928 06:43:48.956475 4285 raft_consensus.cc:2229] T 9a62f245f09345b499afc5f8044ead6b P dbf566031ae74f58997c51cda8755a83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:43:48.956509 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 3845
I20260928 06:43:48.956642 4285 raft_consensus.cc:2258] T 9a62f245f09345b499afc5f8044ead6b P dbf566031ae74f58997c51cda8755a83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:43:48.956892 4283 ts_tablet_manager.cc:1905] T 9a62f245f09345b499afc5f8044ead6b P e8d96ce4e2fe40f6b88d42ad880ea90f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:43:48.958571 4283 ts_tablet_manager.cc:1918] T 9a62f245f09345b499afc5f8044ead6b P e8d96ce4e2fe40f6b88d42ad880ea90f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:43:48.958659 4283 log.cc:1196] T 9a62f245f09345b499afc5f8044ead6b P e8d96ce4e2fe40f6b88d42ad880ea90f: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-2/wal/wals/9a62f245f09345b499afc5f8044ead6b
I20260928 06:43:48.958953 4283 ts_tablet_manager.cc:1939] T 9a62f245f09345b499afc5f8044ead6b P e8d96ce4e2fe40f6b88d42ad880ea90f: Deleting consensus metadata
I20260928 06:43:48.959196 4284 tablet_replica.cc:330] T 9a62f245f09345b499afc5f8044ead6b P 17ba4c282317409e887de72b420833c5: stopping tablet replica
I20260928 06:43:48.959337 3766 catalog_manager.cc:4841] TS e8d96ce4e2fe40f6b88d42ad880ea90f (127.18.211.67:40369): tablet 9a62f245f09345b499afc5f8044ead6b (table test-table [id=586027b91786425480e0320d28756a24]) successfully deleted
I20260928 06:43:48.959365 4284 raft_consensus.cc:2229] T 9a62f245f09345b499afc5f8044ead6b P 17ba4c282317409e887de72b420833c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:43:48.959492 4284 raft_consensus.cc:2258] T 9a62f245f09345b499afc5f8044ead6b P 17ba4c282317409e887de72b420833c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:43:48.959946 4284 ts_tablet_manager.cc:1905] T 9a62f245f09345b499afc5f8044ead6b P 17ba4c282317409e887de72b420833c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:43:48.961580 4284 ts_tablet_manager.cc:1918] T 9a62f245f09345b499afc5f8044ead6b P 17ba4c282317409e887de72b420833c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:43:48.961658 4284 log.cc:1196] T 9a62f245f09345b499afc5f8044ead6b P 17ba4c282317409e887de72b420833c5: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790577602522428-19277-0/minicluster-data/ts-1/wal/wals/9a62f245f09345b499afc5f8044ead6b
W20260928 06:43:48.961881 3766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.211.65:34769 (error 108)
I20260928 06:43:48.961905 4284 ts_tablet_manager.cc:1939] T 9a62f245f09345b499afc5f8044ead6b P 17ba4c282317409e887de72b420833c5: Deleting consensus metadata
I20260928 06:43:48.962203 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 3979
I20260928 06:43:48.962332 3766 catalog_manager.cc:4841] TS 17ba4c282317409e887de72b420833c5 (127.18.211.66:46233): tablet 9a62f245f09345b499afc5f8044ead6b (table test-table [id=586027b91786425480e0320d28756a24]) successfully deleted
I20260928 06:43:48.962793 4256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:48.962288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:34769 (local address 127.0.0.1:42376)
0928 06:43:48.962707 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:48.962758 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.65:34769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394}
W20260928 06:43:48.966981 3766 catalog_manager.cc:4584] TS dbf566031ae74f58997c51cda8755a83 (127.18.211.65:34769): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9a62f245f09345b499afc5f8044ead6b: Network error: Client connection negotiation failed: client connection to 127.18.211.65:34769: connect: Connection refused (error 111)
I20260928 06:43:48.967232 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 4113
I20260928 06:43:48.972010 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 3750
2026-09-28T06:43:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2094 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T06:43:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:43:48Z Disabled control of system clock
I20260928 06:43:48.997926 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:46355
--webserver_interface=127.18.211.126
--webserver_port=0
--builtin_ntp_servers=127.18.211.84:44557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:46355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:43:49.075693 4290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:43:49.075908 4290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:43:49.075928 4290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:43:49.077066 4290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:43:49.077107 4290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:43:49.077118 4290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:43:49.077129 4290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:43:49.077140 4290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:43:49.078457 4290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:44557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:46355
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:46355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:43:49.078675 4290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:43:49.078944 4290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:43:49.083619 4296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:49.083613 4298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:49.083652 4290 server_base.cc:1034] running on GCE node
W20260928 06:43:49.083613 4295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:49.084062 4290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:43:49.084375 4290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:43:49.085539 4290 hybrid_clock.cc:648] HybridClock initialized: now 1790577829085527 us; error 31 us; skew 500 ppm
I20260928 06:43:49.086872 4290 webserver.cc:460] Webserver started at http://127.18.211.126:39547/ using document root <none> and password file <none>
I20260928 06:43:49.087113 4290 fs_manager.cc:359] Metadata directory not provided
I20260928 06:43:49.087173 4290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:43:49.087301 4290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:43:49.088374 4290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "bc5f983b93984e7db71366cbdb7f78fd"
format_stamp: "Formatted at 2026-09-28 06:43:49 on dist-test-slave-b28r"
server_key: "7b38ae1ca073469da3c83a275d74fdc7"
server_key_iv: "cba74aac76f65695fee6a3206a0ad702"
server_key_version: "encryptionkey@0"
I20260928 06:43:49.088698 4290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "bc5f983b93984e7db71366cbdb7f78fd"
format_stamp: "Formatted at 2026-09-28 06:43:49 on dist-test-slave-b28r"
server_key: "7b38ae1ca073469da3c83a275d74fdc7"
server_key_iv: "cba74aac76f65695fee6a3206a0ad702"
server_key_version: "encryptionkey@0"
I20260928 06:43:49.089897 4290 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:43:49.090659 4304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:49.090809 4290 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:43:49.090891 4290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "bc5f983b93984e7db71366cbdb7f78fd"
format_stamp: "Formatted at 2026-09-28 06:43:49 on dist-test-slave-b28r"
server_key: "7b38ae1ca073469da3c83a275d74fdc7"
server_key_iv: "cba74aac76f65695fee6a3206a0ad702"
server_key_version: "encryptionkey@0"
I20260928 06:43:49.090958 4290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:43:49.104288 4290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:43:49.104578 4290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:43:49.104700 4290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:43:49.108438 4290 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:46355
I20260928 06:43:49.108481 4357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:46355 every 8 connection(s)
I20260928 06:43:49.108800 4290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:43:49.109697 4358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:43:49.111908 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 4290
I20260928 06:43:49.112011 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:43:49.112015 4358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc5f983b93984e7db71366cbdb7f78fd: Bootstrap starting.
I20260928 06:43:49.112196 19277 external_mini_cluster.cc:1428] Setting key 511284368a596cb789e2100d775ed7ed
I20260928 06:43:49.112687 4358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc5f983b93984e7db71366cbdb7f78fd: Neither blocks nor log segments found. Creating new log.
I20260928 06:43:49.113063 4358 log.cc:824] T 00000000000000000000000000000000 P bc5f983b93984e7db71366cbdb7f78fd: Log is configured to *not* fsync() on all Append() calls
I20260928 06:43:49.113811 4358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc5f983b93984e7db71366cbdb7f78fd: No bootstrap required, opened a new log
I20260928 06:43:49.116052 4358 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc5f983b93984e7db71366cbdb7f78fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5f983b93984e7db71366cbdb7f78fd" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 46355 } }
I20260928 06:43:49.116211 4358 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc5f983b93984e7db71366cbdb7f78fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:43:49.116247 4358 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc5f983b93984e7db71366cbdb7f78fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc5f983b93984e7db71366cbdb7f78fd, State: Initialized, Role: FOLLOWER
I20260928 06:43:49.116362 4358 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc5f983b93984e7db71366cbdb7f78fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5f983b93984e7db71366cbdb7f78fd" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 46355 } }
I20260928 06:43:49.116434 4358 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc5f983b93984e7db71366cbdb7f78fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:43:49.116461 4358 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc5f983b93984e7db71366cbdb7f78fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:43:49.116503 4358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc5f983b93984e7db71366cbdb7f78fd [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:43:49.117228 4358 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc5f983b93984e7db71366cbdb7f78fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5f983b93984e7db71366cbdb7f78fd" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 46355 } }
I20260928 06:43:49.117344 4358 leader_election.cc:302] T 00000000000000000000000000000000 P bc5f983b93984e7db71366cbdb7f78fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc5f983b93984e7db71366cbdb7f78fd; no voters:
I20260928 06:43:49.117513 4358 leader_election.cc:288] T 00000000000000000000000000000000 P bc5f983b93984e7db71366cbdb7f78fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:43:49.117563 4363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc5f983b93984e7db71366cbdb7f78fd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:43:49.117697 4363 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc5f983b93984e7db71366cbdb7f78fd [term 1 LEADER]: Becoming Leader. State: Replica: bc5f983b93984e7db71366cbdb7f78fd, State: Running, Role: LEADER
I20260928 06:43:49.117831 4363 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc5f983b93984e7db71366cbdb7f78fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5f983b93984e7db71366cbdb7f78fd" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 46355 } }
I20260928 06:43:49.117882 4358 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc5f983b93984e7db71366cbdb7f78fd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:43:49.118327 4361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:49.112679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54558 (local address 127.18.211.126:46355)
0928 06:43:49.112911 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:49.112921 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:49.112957 (+ 36us) server_negotiation.cc:407] Connection header received
0928 06:43:49.113463 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:49.113473 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:49.113552 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:49.113703 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:49.113945 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.115367 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.116157 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.116757 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.116882 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:49.117593 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:49.117597 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:49.117597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:43:49.117807 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:49.117853 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:43:49.117868 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:49.117960 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:49.118051 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 06:43:49.118136 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":91,"threads_started":1}
I20260928 06:43:49.119124 4365 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc5f983b93984e7db71366cbdb7f78fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc5f983b93984e7db71366cbdb7f78fd. Latest consensus state: current_term: 1 leader_uuid: "bc5f983b93984e7db71366cbdb7f78fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5f983b93984e7db71366cbdb7f78fd" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 46355 } } }
I20260928 06:43:49.119213 4365 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc5f983b93984e7db71366cbdb7f78fd [sys.catalog]: This master's current role is: LEADER
I20260928 06:43:49.119576 4364 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc5f983b93984e7db71366cbdb7f78fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc5f983b93984e7db71366cbdb7f78fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5f983b93984e7db71366cbdb7f78fd" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 46355 } } }
I20260928 06:43:49.119647 4364 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc5f983b93984e7db71366cbdb7f78fd [sys.catalog]: This master's current role is: LEADER
W20260928 06:43:49.119647 4377 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc5f983b93984e7db71366cbdb7f78fd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:43:49.119807 4377 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:43:49.119930 4378 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:43:49.120092 4378 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:43:49.121443 4378 catalog_manager.cc:1347] Generated new cluster ID: 42d80d751292423889cf6dded382472f
I20260928 06:43:49.121496 4378 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:43:49.145583 4378 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:43:49.146337 4378 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:43:49.153352 4378 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc5f983b93984e7db71366cbdb7f78fd: Generated new TSK 0
I20260928 06:43:49.153560 4378 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:43:49.157274 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.65:0
--local_ip_for_outbound_sockets=127.18.211.65
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:46355
--builtin_ntp_servers=127.18.211.84:44557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:43:49.237286 4382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:43:49.237489 4382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:43:49.237519 4382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:43:49.238683 4382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:43:49.238723 4382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:43:49.238757 4382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.65
I20260928 06:43:49.240087 4382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:44557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:46355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.65
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:43:49.240331 4382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:43:49.240617 4382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:43:49.245445 4388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:49.245494 4390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:49.245500 4382 server_base.cc:1034] running on GCE node
W20260928 06:43:49.245464 4387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:49.245910 4382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:43:49.246158 4382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:43:49.247313 4382 hybrid_clock.cc:648] HybridClock initialized: now 1790577829247316 us; error 55 us; skew 500 ppm
I20260928 06:43:49.248570 4382 webserver.cc:460] Webserver started at http://127.18.211.65:37547/ using document root <none> and password file <none>
I20260928 06:43:49.248806 4382 fs_manager.cc:359] Metadata directory not provided
I20260928 06:43:49.248853 4382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:43:49.248975 4382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:43:49.250001 4382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-0/data/instance:
uuid: "41313ba335d24193b61537014b76dde3"
format_stamp: "Formatted at 2026-09-28 06:43:49 on dist-test-slave-b28r"
server_key: "24a5065d123b4f4cf316b6b69e5f80fc"
server_key_iv: "078041768631ba80429431779818cdca"
server_key_version: "encryptionkey@0"
I20260928 06:43:49.250319 4382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance:
uuid: "41313ba335d24193b61537014b76dde3"
format_stamp: "Formatted at 2026-09-28 06:43:49 on dist-test-slave-b28r"
server_key: "24a5065d123b4f4cf316b6b69e5f80fc"
server_key_iv: "078041768631ba80429431779818cdca"
server_key_version: "encryptionkey@0"
I20260928 06:43:49.251595 4382 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:43:49.252282 4396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:49.252418 4382 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:43:49.252491 4382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-0/wal
uuid: "41313ba335d24193b61537014b76dde3"
format_stamp: "Formatted at 2026-09-28 06:43:49 on dist-test-slave-b28r"
server_key: "24a5065d123b4f4cf316b6b69e5f80fc"
server_key_iv: "078041768631ba80429431779818cdca"
server_key_version: "encryptionkey@0"
I20260928 06:43:49.252553 4382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:43:49.274863 4382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:43:49.275192 4382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:43:49.275321 4382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:43:49.275560 4382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:43:49.275889 4382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:43:49.275923 4382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:49.275950 4382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:43:49.275969 4382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:49.281495 4382 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.65:46451
I20260928 06:43:49.281561 4510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.65:46451 every 8 connection(s)
I20260928 06:43:49.281857 4382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
I20260928 06:43:49.282343 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 4382
I20260928 06:43:49.282420 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance
I20260928 06:43:49.282603 19277 external_mini_cluster.cc:1428] Setting key 0e8f2c7738116566d93c9c9cb475aad6
I20260928 06:43:49.283664 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.66:0
--local_ip_for_outbound_sockets=127.18.211.66
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:46355
--builtin_ntp_servers=127.18.211.84:44557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:43:49.287863 4361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:49.283127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:51935 (local address 127.18.211.126:46355)
0928 06:43:49.283174 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:49.283176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:49.283486 (+ 310us) server_negotiation.cc:407] Connection header received
0928 06:43:49.284033 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:49.284035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:49.284047 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:49.284113 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:49.284772 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.285288 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.286351 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.286647 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.286720 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:49.287590 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:49.287591 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:49.287592 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:43:49.287597 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:49.287615 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:43:49.287619 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:49.287683 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:49.287746 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 06:43:49.287770 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:43:49.288057 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:49.283152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:46355 (local address 127.18.211.65:51935)
0928 06:43:49.283349 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:49.283396 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:49.283943 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:49.284169 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:49.284172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:49.284179 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:43:49.284681 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:49.284687 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.285374 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.285375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:49.286267 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:49.286270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.286770 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.286771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:49.286902 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:49.287457 (+ 555us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:43:49.287458 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:43:49.287487 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:43:49.287511 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:43:49.287682 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:43:49.287685 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:43:49.287688 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:43:49.287731 (+ 43us) client_negotiation.cc:240] Negotiation successful
0928 06:43:49.287808 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260928 06:43:49.288524 4511 heartbeater.cc:347] Connected to a master server at 127.18.211.126:46355
I20260928 06:43:49.288604 4511 heartbeater.cc:464] Registering TS with master...
I20260928 06:43:49.288779 4511 heartbeater.cc:511] Master 127.18.211.126:46355 requested a full tablet report, sending...
I20260928 06:43:49.289364 4321 ts_manager.cc:194] Registered new tserver with Master: 41313ba335d24193b61537014b76dde3 (127.18.211.65:46451)
I20260928 06:43:49.289964 4321 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.211.65:51935
W20260928 06:43:49.364634 4514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:43:49.364843 4514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:43:49.364873 4514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:43:49.365983 4514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:43:49.366021 4514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:43:49.366053 4514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.66
I20260928 06:43:49.367432 4514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:44557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:46355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.66
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:43:49.367651 4514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:43:49.367933 4514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:43:49.372534 4520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:49.372566 4519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:49.372591 4522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:49.372814 4514 server_base.cc:1034] running on GCE node
I20260928 06:43:49.373004 4514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:43:49.373210 4514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:43:49.374344 4514 hybrid_clock.cc:648] HybridClock initialized: now 1790577829374331 us; error 29 us; skew 500 ppm
I20260928 06:43:49.375635 4514 webserver.cc:460] Webserver started at http://127.18.211.66:39501/ using document root <none> and password file <none>
I20260928 06:43:49.375844 4514 fs_manager.cc:359] Metadata directory not provided
I20260928 06:43:49.375892 4514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:43:49.376016 4514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:43:49.377043 4514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-1/data/instance:
uuid: "0e676019fbd1414ba35874f1934bf852"
format_stamp: "Formatted at 2026-09-28 06:43:49 on dist-test-slave-b28r"
server_key: "3588d18ffdbd084e5e3896d0fd75f275"
server_key_iv: "15c1db6fc9d2e2cb14b9354d829eb7cc"
server_key_version: "encryptionkey@0"
I20260928 06:43:49.377462 4514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance:
uuid: "0e676019fbd1414ba35874f1934bf852"
format_stamp: "Formatted at 2026-09-28 06:43:49 on dist-test-slave-b28r"
server_key: "3588d18ffdbd084e5e3896d0fd75f275"
server_key_iv: "15c1db6fc9d2e2cb14b9354d829eb7cc"
server_key_version: "encryptionkey@0"
I20260928 06:43:49.378799 4514 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:43:49.379555 4528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:49.379702 4514 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:49.379782 4514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-1/wal
uuid: "0e676019fbd1414ba35874f1934bf852"
format_stamp: "Formatted at 2026-09-28 06:43:49 on dist-test-slave-b28r"
server_key: "3588d18ffdbd084e5e3896d0fd75f275"
server_key_iv: "15c1db6fc9d2e2cb14b9354d829eb7cc"
server_key_version: "encryptionkey@0"
I20260928 06:43:49.379840 4514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:43:49.397270 4514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:43:49.397575 4514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:43:49.397709 4514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:43:49.397910 4514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:43:49.398247 4514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:43:49.398291 4514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:49.398319 4514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:43:49.398343 4514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:49.403729 4514 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.66:40811
I20260928 06:43:49.403795 4642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.66:40811 every 8 connection(s)
I20260928 06:43:49.404091 4514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
I20260928 06:43:49.408449 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 4514
I20260928 06:43:49.408530 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance
I20260928 06:43:49.408718 19277 external_mini_cluster.cc:1428] Setting key 1fa2fba5d79722647412bcfad75fd85f
I20260928 06:43:49.409209 4361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:49.404878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:44573 (local address 127.18.211.126:46355)
0928 06:43:49.404943 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:49.404945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:49.405163 (+ 218us) server_negotiation.cc:407] Connection header received
0928 06:43:49.405683 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:49.405684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:49.405694 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:49.405749 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:49.406372 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.406861 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.407893 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.408073 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.408120 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:49.408946 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:49.408947 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:49.408947 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:43:49.408952 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:49.408967 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:43:49.408971 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:49.409011 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:49.409105 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 06:43:49.409130 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 06:43:49.409435 4645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:49.404837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:46355 (local address 127.18.211.66:44573)
0928 06:43:49.405048 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:49.405083 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:49.405592 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:49.405823 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:49.405826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:49.405834 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:43:49.406311 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:49.406315 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.406948 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.406949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:49.407820 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:49.407823 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.408129 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.408130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:49.408234 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:49.408792 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:43:49.408792 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:43:49.408832 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:43:49.408861 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:43:49.409025 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:43:49.409032 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:43:49.409035 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:43:49.409105 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 06:43:49.409209 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":81,"threads_started":1}
I20260928 06:43:49.409588 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.67:0
--local_ip_for_outbound_sockets=127.18.211.67
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:46355
--builtin_ntp_servers=127.18.211.84:44557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:43:49.409912 4643 heartbeater.cc:347] Connected to a master server at 127.18.211.126:46355
I20260928 06:43:49.410009 4643 heartbeater.cc:464] Registering TS with master...
I20260928 06:43:49.410194 4643 heartbeater.cc:511] Master 127.18.211.126:46355 requested a full tablet report, sending...
I20260928 06:43:49.410542 4321 ts_manager.cc:194] Registered new tserver with Master: 0e676019fbd1414ba35874f1934bf852 (127.18.211.66:40811)
I20260928 06:43:49.410979 4321 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.211.66:44573
W20260928 06:43:49.489578 4646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:43:49.489768 4646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:43:49.489789 4646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:43:49.490900 4646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:43:49.490938 4646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:43:49.490962 4646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.67
I20260928 06:43:49.492302 4646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:44557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:46355
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.67
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:43:49.492488 4646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:43:49.492733 4646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:43:49.497565 4654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:49.497571 4651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:49.497741 4646 server_base.cc:1034] running on GCE node
W20260928 06:43:49.497649 4652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:49.498015 4646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:43:49.498258 4646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:43:49.499397 4646 hybrid_clock.cc:648] HybridClock initialized: now 1790577829499376 us; error 36 us; skew 500 ppm
I20260928 06:43:49.500672 4646 webserver.cc:460] Webserver started at http://127.18.211.67:35397/ using document root <none> and password file <none>
I20260928 06:43:49.500896 4646 fs_manager.cc:359] Metadata directory not provided
I20260928 06:43:49.500944 4646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:43:49.501049 4646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:43:49.502061 4646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-2/data/instance:
uuid: "d4908192cd2f495b89bf5e33b5553a32"
format_stamp: "Formatted at 2026-09-28 06:43:49 on dist-test-slave-b28r"
server_key: "f51af0ddec5bf9e1fce2acf159e75605"
server_key_iv: "22ac17a8bde29bb17b3057518c8c7c79"
server_key_version: "encryptionkey@0"
I20260928 06:43:49.502384 4646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance:
uuid: "d4908192cd2f495b89bf5e33b5553a32"
format_stamp: "Formatted at 2026-09-28 06:43:49 on dist-test-slave-b28r"
server_key: "f51af0ddec5bf9e1fce2acf159e75605"
server_key_iv: "22ac17a8bde29bb17b3057518c8c7c79"
server_key_version: "encryptionkey@0"
I20260928 06:43:49.503624 4646 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:43:49.504292 4660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:49.504423 4646 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:43:49.504510 4646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-2/wal
uuid: "d4908192cd2f495b89bf5e33b5553a32"
format_stamp: "Formatted at 2026-09-28 06:43:49 on dist-test-slave-b28r"
server_key: "f51af0ddec5bf9e1fce2acf159e75605"
server_key_iv: "22ac17a8bde29bb17b3057518c8c7c79"
server_key_version: "encryptionkey@0"
I20260928 06:43:49.504571 4646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:43:49.520073 4646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:43:49.520357 4646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:43:49.520485 4646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:43:49.520706 4646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:43:49.521016 4646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:43:49.521049 4646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:49.521068 4646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:43:49.521080 4646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:49.526484 4646 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.67:34857
I20260928 06:43:49.526538 4774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.67:34857 every 8 connection(s)
I20260928 06:43:49.526837 4646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
I20260928 06:43:49.531848 4361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:49.527596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:47357 (local address 127.18.211.126:46355)
0928 06:43:49.527666 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:49.527668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:49.527854 (+ 186us) server_negotiation.cc:407] Connection header received
0928 06:43:49.528383 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:49.528385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:49.528394 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:49.528443 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:49.529069 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.529544 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.530531 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.530745 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.530806 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:49.531628 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:49.531629 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:49.531629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:43:49.531634 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:49.531647 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:43:49.531651 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:49.531694 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:49.531766 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 06:43:49.531788 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 06:43:49.532033 4777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:49.527552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:46355 (local address 127.18.211.67:47357)
0928 06:43:49.527735 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:49.527771 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:49.528293 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:49.528507 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:49.528510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:49.528519 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:43:49.529005 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:49.529009 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.529624 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.529625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:49.530433 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:49.530435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.530828 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.530829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:49.530951 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:49.531488 (+ 537us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:43:49.531488 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:43:49.531518 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:43:49.531541 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:43:49.531704 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:43:49.531706 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:43:49.531709 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:43:49.531754 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 06:43:49.531833 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
I20260928 06:43:49.532532 4775 heartbeater.cc:347] Connected to a master server at 127.18.211.126:46355
I20260928 06:43:49.532618 4775 heartbeater.cc:464] Registering TS with master...
I20260928 06:43:49.532781 4775 heartbeater.cc:511] Master 127.18.211.126:46355 requested a full tablet report, sending...
I20260928 06:43:49.533102 4321 ts_manager.cc:194] Registered new tserver with Master: d4908192cd2f495b89bf5e33b5553a32 (127.18.211.67:34857)
I20260928 06:43:49.533455 4321 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.211.67:47357
I20260928 06:43:49.534425 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 4646
I20260928 06:43:49.534513 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance
I20260928 06:43:49.534696 19277 external_mini_cluster.cc:1428] Setting key df30daf7c671d3cbd6c886db73cd7c2f
I20260928 06:43:49.535622 19277 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:43:49.539582 4361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:49.536931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54562 (local address 127.18.211.126:46355)
0928 06:43:49.536978 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:49.536980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:49.537074 (+ 94us) server_negotiation.cc:407] Connection header received
0928 06:43:49.537112 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:49.537113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:49.537120 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:49.537162 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:49.537298 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.537761 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.538518 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.538707 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.538776 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:49.539414 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:49.539415 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:49.539416 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:43:49.539420 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:49.539433 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:43:49.539436 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:49.539476 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:49.539510 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 06:43:49.539529 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:43:49.541052 4321 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:43:49.542565 4321 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:43:49.548668 4783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:49.544762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:34857 (local address 127.0.0.1:37664)
0928 06:43:49.544888 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:49.544929 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:49.544991 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:49.545383 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:49.545383 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:49.545385 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:43:49.545419 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:49.545420 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.546671 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.546672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:49.547748 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:49.547750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.548394 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.548395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:49.548470 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:49.548484 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 06:43:49.548542 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 06:43:49.548593 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
I20260928 06:43:49.548681 4788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:49.544857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37664 (local address 127.18.211.67:34857)
0928 06:43:49.545192 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:49.545196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:49.545205 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:43:49.545237 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:49.545239 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:49.545254 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:49.545326 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:43:49.545478 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.546585 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.547827 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.548318 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.548381 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:49.548472 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:49.548568 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 06:43:49.548601 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":68,"threads_started":1}
I20260928 06:43:49.548904 4784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:49.544544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:40811 (local address 127.0.0.1:43480)
0928 06:43:49.544974 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:49.544984 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:49.545073 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:49.545303 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:49.545304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:49.545305 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:43:49.545352 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:49.545354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.546394 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.546395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:49.547640 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:49.547643 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.548619 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.548620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:49.548700 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:49.548704 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:43:49.548753 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 06:43:49.548807 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":64,"thread_start_us":72,"threads_started":1}
I20260928 06:43:49.548988 4787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:49.544844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60274 (local address 127.18.211.65:46451)
0928 06:43:49.545039 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:49.545044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:49.545054 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:43:49.545091 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:49.545093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:49.545107 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:49.545182 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:43:49.545415 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.546623 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.547666 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.548425 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.548495 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:49.548583 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:49.548854 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 06:43:49.548905 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":78,"threads_started":1}
I20260928 06:43:49.548913 4786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:49.544579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:46451 (local address 127.0.0.1:60274)
0928 06:43:49.544882 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:49.544929 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:49.544991 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:49.545198 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:49.545200 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:49.545203 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:43:49.545280 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:49.545282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.546703 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.546704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:49.547592 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:49.547595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.548621 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.548622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:49.548700 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:49.548705 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:43:49.548757 (+ 52us) client_negotiation.cc:240] Negotiation successful
0928 06:43:49.548805 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":4,"thread_start_us":102,"threads_started":1}
I20260928 06:43:49.549158 4785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:49.544663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43480 (local address 127.18.211.66:40811)
0928 06:43:49.544807 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:49.544812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:49.545044 (+ 232us) server_negotiation.cc:407] Connection header received
0928 06:43:49.545156 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:49.545159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:49.545174 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:49.545240 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:43:49.545564 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.546310 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.547707 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.548425 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.548495 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:49.548582 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:49.548990 (+ 408us) server_negotiation.cc:299] Negotiation successful
0928 06:43:49.549078 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":73,"threads_started":1}
I20260928 06:43:49.549691 4708 tablet_service.cc:1467] Processing CreateTablet for tablet 5e9a29e5ee5c45e38781020241d7ff0a (DEFAULT_TABLE table=test-table [id=d111d4ef9c7c420e9db31789f62a4884]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:43:49.549818 4444 tablet_service.cc:1467] Processing CreateTablet for tablet 5e9a29e5ee5c45e38781020241d7ff0a (DEFAULT_TABLE table=test-table [id=d111d4ef9c7c420e9db31789f62a4884]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:43:49.550063 4708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e9a29e5ee5c45e38781020241d7ff0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:43:49.549980 4576 tablet_service.cc:1467] Processing CreateTablet for tablet 5e9a29e5ee5c45e38781020241d7ff0a (DEFAULT_TABLE table=test-table [id=d111d4ef9c7c420e9db31789f62a4884]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:43:49.550096 4444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e9a29e5ee5c45e38781020241d7ff0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:43:49.550230 4576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e9a29e5ee5c45e38781020241d7ff0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:43:49.552812 4789 tablet_bootstrap.cc:492] T 5e9a29e5ee5c45e38781020241d7ff0a P 0e676019fbd1414ba35874f1934bf852: Bootstrap starting.
I20260928 06:43:49.553285 4790 tablet_bootstrap.cc:492] T 5e9a29e5ee5c45e38781020241d7ff0a P d4908192cd2f495b89bf5e33b5553a32: Bootstrap starting.
I20260928 06:43:49.553522 4789 tablet_bootstrap.cc:654] T 5e9a29e5ee5c45e38781020241d7ff0a P 0e676019fbd1414ba35874f1934bf852: Neither blocks nor log segments found. Creating new log.
I20260928 06:43:49.553503 4791 tablet_bootstrap.cc:492] T 5e9a29e5ee5c45e38781020241d7ff0a P 41313ba335d24193b61537014b76dde3: Bootstrap starting.
I20260928 06:43:49.553854 4789 log.cc:824] T 5e9a29e5ee5c45e38781020241d7ff0a P 0e676019fbd1414ba35874f1934bf852: Log is configured to *not* fsync() on all Append() calls
I20260928 06:43:49.554136 4791 tablet_bootstrap.cc:654] T 5e9a29e5ee5c45e38781020241d7ff0a P 41313ba335d24193b61537014b76dde3: Neither blocks nor log segments found. Creating new log.
I20260928 06:43:49.554412 4791 log.cc:824] T 5e9a29e5ee5c45e38781020241d7ff0a P 41313ba335d24193b61537014b76dde3: Log is configured to *not* fsync() on all Append() calls
I20260928 06:43:49.554512 4789 tablet_bootstrap.cc:492] T 5e9a29e5ee5c45e38781020241d7ff0a P 0e676019fbd1414ba35874f1934bf852: No bootstrap required, opened a new log
I20260928 06:43:49.554572 4789 ts_tablet_manager.cc:1397] T 5e9a29e5ee5c45e38781020241d7ff0a P 0e676019fbd1414ba35874f1934bf852: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:43:49.555155 4790 tablet_bootstrap.cc:654] T 5e9a29e5ee5c45e38781020241d7ff0a P d4908192cd2f495b89bf5e33b5553a32: Neither blocks nor log segments found. Creating new log.
I20260928 06:43:49.555160 4791 tablet_bootstrap.cc:492] T 5e9a29e5ee5c45e38781020241d7ff0a P 41313ba335d24193b61537014b76dde3: No bootstrap required, opened a new log
I20260928 06:43:49.555231 4791 ts_tablet_manager.cc:1397] T 5e9a29e5ee5c45e38781020241d7ff0a P 41313ba335d24193b61537014b76dde3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:43:49.555480 4790 log.cc:824] T 5e9a29e5ee5c45e38781020241d7ff0a P d4908192cd2f495b89bf5e33b5553a32: Log is configured to *not* fsync() on all Append() calls
I20260928 06:43:49.556249 4790 tablet_bootstrap.cc:492] T 5e9a29e5ee5c45e38781020241d7ff0a P d4908192cd2f495b89bf5e33b5553a32: No bootstrap required, opened a new log
I20260928 06:43:49.556313 4790 ts_tablet_manager.cc:1397] T 5e9a29e5ee5c45e38781020241d7ff0a P d4908192cd2f495b89bf5e33b5553a32: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260928 06:43:49.557242 4789 raft_consensus.cc:348] T 5e9a29e5ee5c45e38781020241d7ff0a P 0e676019fbd1414ba35874f1934bf852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e676019fbd1414ba35874f1934bf852" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 40811 } } peers { permanent_uuid: "d4908192cd2f495b89bf5e33b5553a32" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34857 } } peers { permanent_uuid: "41313ba335d24193b61537014b76dde3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 46451 } }
I20260928 06:43:49.557471 4789 raft_consensus.cc:374] T 5e9a29e5ee5c45e38781020241d7ff0a P 0e676019fbd1414ba35874f1934bf852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:43:49.557534 4789 raft_consensus.cc:729] T 5e9a29e5ee5c45e38781020241d7ff0a P 0e676019fbd1414ba35874f1934bf852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e676019fbd1414ba35874f1934bf852, State: Initialized, Role: FOLLOWER
I20260928 06:43:49.557703 4789 consensus_queue.cc:260] T 5e9a29e5ee5c45e38781020241d7ff0a P 0e676019fbd1414ba35874f1934bf852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e676019fbd1414ba35874f1934bf852" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 40811 } } peers { permanent_uuid: "d4908192cd2f495b89bf5e33b5553a32" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34857 } } peers { permanent_uuid: "41313ba335d24193b61537014b76dde3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 46451 } }
I20260928 06:43:49.557687 4791 raft_consensus.cc:348] T 5e9a29e5ee5c45e38781020241d7ff0a P 41313ba335d24193b61537014b76dde3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e676019fbd1414ba35874f1934bf852" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 40811 } } peers { permanent_uuid: "d4908192cd2f495b89bf5e33b5553a32" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34857 } } peers { permanent_uuid: "41313ba335d24193b61537014b76dde3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 46451 } }
I20260928 06:43:49.557826 4791 raft_consensus.cc:374] T 5e9a29e5ee5c45e38781020241d7ff0a P 41313ba335d24193b61537014b76dde3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:43:49.557884 4791 raft_consensus.cc:729] T 5e9a29e5ee5c45e38781020241d7ff0a P 41313ba335d24193b61537014b76dde3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41313ba335d24193b61537014b76dde3, State: Initialized, Role: FOLLOWER
I20260928 06:43:49.558035 4789 ts_tablet_manager.cc:1428] T 5e9a29e5ee5c45e38781020241d7ff0a P 0e676019fbd1414ba35874f1934bf852: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:43:49.558007 4791 consensus_queue.cc:260] T 5e9a29e5ee5c45e38781020241d7ff0a P 41313ba335d24193b61537014b76dde3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e676019fbd1414ba35874f1934bf852" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 40811 } } peers { permanent_uuid: "d4908192cd2f495b89bf5e33b5553a32" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34857 } } peers { permanent_uuid: "41313ba335d24193b61537014b76dde3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 46451 } }
I20260928 06:43:49.558166 4643 heartbeater.cc:503] Master 127.18.211.126:46355 was elected leader, sending a full tablet report...
I20260928 06:43:49.558254 4791 ts_tablet_manager.cc:1428] T 5e9a29e5ee5c45e38781020241d7ff0a P 41313ba335d24193b61537014b76dde3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:43:49.558282 4511 heartbeater.cc:503] Master 127.18.211.126:46355 was elected leader, sending a full tablet report...
I20260928 06:43:49.558681 4790 raft_consensus.cc:348] T 5e9a29e5ee5c45e38781020241d7ff0a P d4908192cd2f495b89bf5e33b5553a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e676019fbd1414ba35874f1934bf852" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 40811 } } peers { permanent_uuid: "d4908192cd2f495b89bf5e33b5553a32" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34857 } } peers { permanent_uuid: "41313ba335d24193b61537014b76dde3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 46451 } }
I20260928 06:43:49.558831 4790 raft_consensus.cc:374] T 5e9a29e5ee5c45e38781020241d7ff0a P d4908192cd2f495b89bf5e33b5553a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:43:49.558868 4790 raft_consensus.cc:729] T 5e9a29e5ee5c45e38781020241d7ff0a P d4908192cd2f495b89bf5e33b5553a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4908192cd2f495b89bf5e33b5553a32, State: Initialized, Role: FOLLOWER
I20260928 06:43:49.559000 4790 consensus_queue.cc:260] T 5e9a29e5ee5c45e38781020241d7ff0a P d4908192cd2f495b89bf5e33b5553a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e676019fbd1414ba35874f1934bf852" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 40811 } } peers { permanent_uuid: "d4908192cd2f495b89bf5e33b5553a32" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34857 } } peers { permanent_uuid: "41313ba335d24193b61537014b76dde3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 46451 } }
I20260928 06:43:49.559283 4790 ts_tablet_manager.cc:1428] T 5e9a29e5ee5c45e38781020241d7ff0a P d4908192cd2f495b89bf5e33b5553a32: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 06:43:49.559291 4775 heartbeater.cc:503] Master 127.18.211.126:46355 was elected leader, sending a full tablet report...
W20260928 06:43:49.654713 4644 tablet.cc:2369] T 5e9a29e5ee5c45e38781020241d7ff0a P 0e676019fbd1414ba35874f1934bf852: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:43:49.691483 4795 raft_consensus.cc:482] T 5e9a29e5ee5c45e38781020241d7ff0a P 0e676019fbd1414ba35874f1934bf852 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:43:49.691617 4795 raft_consensus.cc:504] T 5e9a29e5ee5c45e38781020241d7ff0a P 0e676019fbd1414ba35874f1934bf852 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e676019fbd1414ba35874f1934bf852" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 40811 } } peers { permanent_uuid: "d4908192cd2f495b89bf5e33b5553a32" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34857 } } peers { permanent_uuid: "41313ba335d24193b61537014b76dde3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 46451 } }
I20260928 06:43:49.691972 4795 leader_election.cc:288] T 5e9a29e5ee5c45e38781020241d7ff0a P 0e676019fbd1414ba35874f1934bf852 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4908192cd2f495b89bf5e33b5553a32 (127.18.211.67:34857), 41313ba335d24193b61537014b76dde3 (127.18.211.65:46451)
I20260928 06:43:49.695371 4645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:49.692149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:46451 (local address 127.18.211.66:46607)
0928 06:43:49.692213 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:49.692227 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:49.692282 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:49.692564 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:49.692565 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:49.692567 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:43:49.692680 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:49.692682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.693347 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.693348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:49.694373 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:49.694377 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.695074 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.695075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:49.695180 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:49.695184 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:43:49.695244 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 06:43:49.695289 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 06:43:49.695377 4787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:49.692374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:46607 (local address 127.18.211.65:46451)
0928 06:43:49.692436 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:49.692438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:49.692446 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:43:49.692475 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:49.692476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:49.692490 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:49.692544 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:43:49.692738 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.693275 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.694474 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.694992 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.695065 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:49.695075 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:49.695250 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 06:43:49.695302 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:43:49.695785 4464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e9a29e5ee5c45e38781020241d7ff0a" candidate_uuid: "0e676019fbd1414ba35874f1934bf852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41313ba335d24193b61537014b76dde3" is_pre_election: true
I20260928 06:43:49.695964 4464 raft_consensus.cc:2454] T 5e9a29e5ee5c45e38781020241d7ff0a P 41313ba335d24193b61537014b76dde3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e676019fbd1414ba35874f1934bf852 in term 0.
I20260928 06:43:49.696195 4532 leader_election.cc:302] T 5e9a29e5ee5c45e38781020241d7ff0a P 0e676019fbd1414ba35874f1934bf852 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e676019fbd1414ba35874f1934bf852, 41313ba335d24193b61537014b76dde3; no voters:
I20260928 06:43:49.696375 4795 raft_consensus.cc:2781] T 5e9a29e5ee5c45e38781020241d7ff0a P 0e676019fbd1414ba35874f1934bf852 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:43:49.696429 4795 raft_consensus.cc:482] T 5e9a29e5ee5c45e38781020241d7ff0a P 0e676019fbd1414ba35874f1934bf852 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:43:49.696450 4795 raft_consensus.cc:3037] T 5e9a29e5ee5c45e38781020241d7ff0a P 0e676019fbd1414ba35874f1934bf852 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:43:49.697269 4795 raft_consensus.cc:504] T 5e9a29e5ee5c45e38781020241d7ff0a P 0e676019fbd1414ba35874f1934bf852 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e676019fbd1414ba35874f1934bf852" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 40811 } } peers { permanent_uuid: "d4908192cd2f495b89bf5e33b5553a32" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34857 } } peers { permanent_uuid: "41313ba335d24193b61537014b76dde3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 46451 } }
I20260928 06:43:49.697427 4795 leader_election.cc:288] T 5e9a29e5ee5c45e38781020241d7ff0a P 0e676019fbd1414ba35874f1934bf852 [CANDIDATE]: Term 1 election: Requested vote from peers d4908192cd2f495b89bf5e33b5553a32 (127.18.211.67:34857), 41313ba335d24193b61537014b76dde3 (127.18.211.65:46451)
I20260928 06:43:49.697690 4788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:49.692299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:41111 (local address 127.18.211.67:34857)
0928 06:43:49.692347 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:49.692349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:49.692458 (+ 109us) server_negotiation.cc:407] Connection header received
0928 06:43:49.692493 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:49.692494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:49.692503 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:49.692550 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:43:49.692684 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.693393 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.694763 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.697304 (+ 2541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.697376 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:49.697384 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:49.697574 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 06:43:49.697621 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:43:49.697897 4798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:49.692151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:34857 (local address 127.18.211.66:41111)
0928 06:43:49.692372 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:49.692395 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:49.692447 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:49.692557 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:49.692558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:49.692559 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:43:49.692630 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:49.692632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.693460 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.693461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:49.694558 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:49.694562 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.697421 (+ 2859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.697422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:49.697493 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:49.697497 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:43:49.697777 (+ 280us) client_negotiation.cc:240] Negotiation successful
0928 06:43:49.697821 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":36,"thread_start_us":101,"threads_started":1}
I20260928 06:43:49.698266 4726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e9a29e5ee5c45e38781020241d7ff0a" candidate_uuid: "0e676019fbd1414ba35874f1934bf852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4908192cd2f495b89bf5e33b5553a32"
I20260928 06:43:49.698246 4728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e9a29e5ee5c45e38781020241d7ff0a" candidate_uuid: "0e676019fbd1414ba35874f1934bf852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4908192cd2f495b89bf5e33b5553a32" is_pre_election: true
I20260928 06:43:49.698441 4728 raft_consensus.cc:2454] T 5e9a29e5ee5c45e38781020241d7ff0a P d4908192cd2f495b89bf5e33b5553a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e676019fbd1414ba35874f1934bf852 in term 0.
I20260928 06:43:49.699164 4464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e9a29e5ee5c45e38781020241d7ff0a" candidate_uuid: "0e676019fbd1414ba35874f1934bf852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41313ba335d24193b61537014b76dde3"
I20260928 06:43:49.699249 4464 raft_consensus.cc:3037] T 5e9a29e5ee5c45e38781020241d7ff0a P 41313ba335d24193b61537014b76dde3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:43:49.699957 4464 raft_consensus.cc:2454] T 5e9a29e5ee5c45e38781020241d7ff0a P 41313ba335d24193b61537014b76dde3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e676019fbd1414ba35874f1934bf852 in term 1.
I20260928 06:43:49.700172 4532 leader_election.cc:302] T 5e9a29e5ee5c45e38781020241d7ff0a P 0e676019fbd1414ba35874f1934bf852 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0e676019fbd1414ba35874f1934bf852, 41313ba335d24193b61537014b76dde3; no voters: d4908192cd2f495b89bf5e33b5553a32
I20260928 06:43:49.700276 4795 raft_consensus.cc:2781] T 5e9a29e5ee5c45e38781020241d7ff0a P 0e676019fbd1414ba35874f1934bf852 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:43:49.700445 4795 raft_consensus.cc:686] T 5e9a29e5ee5c45e38781020241d7ff0a P 0e676019fbd1414ba35874f1934bf852 [term 1 LEADER]: Becoming Leader. State: Replica: 0e676019fbd1414ba35874f1934bf852, State: Running, Role: LEADER
I20260928 06:43:49.700536 4795 consensus_queue.cc:237] T 5e9a29e5ee5c45e38781020241d7ff0a P 0e676019fbd1414ba35874f1934bf852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e676019fbd1414ba35874f1934bf852" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 40811 } } peers { permanent_uuid: "d4908192cd2f495b89bf5e33b5553a32" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34857 } } peers { permanent_uuid: "41313ba335d24193b61537014b76dde3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 46451 } }
I20260928 06:43:49.701381 4321 catalog_manager.cc:5495] T 5e9a29e5ee5c45e38781020241d7ff0a P 0e676019fbd1414ba35874f1934bf852 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e676019fbd1414ba35874f1934bf852 (127.18.211.66). New cstate: current_term: 1 leader_uuid: "0e676019fbd1414ba35874f1934bf852" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e676019fbd1414ba35874f1934bf852" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 40811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4908192cd2f495b89bf5e33b5553a32" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41313ba335d24193b61537014b76dde3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 46451 } health_report { overall_health: UNKNOWN } } }
I20260928 06:43:49.728994 4785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:49.726223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43482 (local address 127.18.211.66:40811)
0928 06:43:49.726268 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:49.726270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:49.726278 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:43:49.726318 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:49.726319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:49.726337 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:49.726392 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:43:49.726532 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.726996 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.728096 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:49.728417 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:49.728487 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:49.728642 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:43:49.728768 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:43:49.728846 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:49.728867 (+ 21us) server_negotiation.cc:299] Negotiation successful
0928 06:43:49.728886 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:43:49.730885 4728 raft_consensus.cc:3037] T 5e9a29e5ee5c45e38781020241d7ff0a P d4908192cd2f495b89bf5e33b5553a32 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:43:49.730988 4464 raft_consensus.cc:1261] T 5e9a29e5ee5c45e38781020241d7ff0a P 41313ba335d24193b61537014b76dde3 [term 1 FOLLOWER]: Refusing update from remote peer 0e676019fbd1414ba35874f1934bf852: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:43:49.731349 4795 consensus_queue.cc:1035] T 5e9a29e5ee5c45e38781020241d7ff0a P 0e676019fbd1414ba35874f1934bf852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41313ba335d24193b61537014b76dde3" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 46451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:43:49.732031 4728 raft_consensus.cc:1261] T 5e9a29e5ee5c45e38781020241d7ff0a P d4908192cd2f495b89bf5e33b5553a32 [term 1 FOLLOWER]: Refusing update from remote peer 0e676019fbd1414ba35874f1934bf852: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:43:49.732302 4795 consensus_queue.cc:1035] T 5e9a29e5ee5c45e38781020241d7ff0a P 0e676019fbd1414ba35874f1934bf852 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4908192cd2f495b89bf5e33b5553a32" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:43:49.736528 4320 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:43:49.736624 4320 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:43:49.737355 4320 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bc5f983b93984e7db71366cbdb7f78fd: Sending DeleteTablet for 3 replicas of tablet 5e9a29e5ee5c45e38781020241d7ff0a
I20260928 06:43:49.737686 4444 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e9a29e5ee5c45e38781020241d7ff0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:43:49 UTC) from {username='slave'} at 127.0.0.1:60274
I20260928 06:43:49.737670 4708 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e9a29e5ee5c45e38781020241d7ff0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:43:49 UTC) from {username='slave'} at 127.0.0.1:37664
I20260928 06:43:49.737941 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 4382
I20260928 06:43:49.737990 4576 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e9a29e5ee5c45e38781020241d7ff0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:43:49 UTC) from {username='slave'} at 127.0.0.1:43480
I20260928 06:43:49.738741 4814 tablet_replica.cc:330] T 5e9a29e5ee5c45e38781020241d7ff0a P 0e676019fbd1414ba35874f1934bf852: stopping tablet replica
I20260928 06:43:49.738835 4814 raft_consensus.cc:2229] T 5e9a29e5ee5c45e38781020241d7ff0a P 0e676019fbd1414ba35874f1934bf852 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:43:49.738919 4814 raft_consensus.cc:2258] T 5e9a29e5ee5c45e38781020241d7ff0a P 0e676019fbd1414ba35874f1934bf852 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:43:49.739328 4814 ts_tablet_manager.cc:1905] T 5e9a29e5ee5c45e38781020241d7ff0a P 0e676019fbd1414ba35874f1934bf852: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:43:49.740495 4812 tablet_replica.cc:330] T 5e9a29e5ee5c45e38781020241d7ff0a P d4908192cd2f495b89bf5e33b5553a32: stopping tablet replica
I20260928 06:43:49.740657 4812 raft_consensus.cc:2229] T 5e9a29e5ee5c45e38781020241d7ff0a P d4908192cd2f495b89bf5e33b5553a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:43:49.740813 4812 raft_consensus.cc:2258] T 5e9a29e5ee5c45e38781020241d7ff0a P d4908192cd2f495b89bf5e33b5553a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:43:49.741103 4814 ts_tablet_manager.cc:1918] T 5e9a29e5ee5c45e38781020241d7ff0a P 0e676019fbd1414ba35874f1934bf852: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:43:49.741173 4814 log.cc:1196] T 5e9a29e5ee5c45e38781020241d7ff0a P 0e676019fbd1414ba35874f1934bf852: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-1/wal/wals/5e9a29e5ee5c45e38781020241d7ff0a
I20260928 06:43:49.741338 4812 ts_tablet_manager.cc:1905] T 5e9a29e5ee5c45e38781020241d7ff0a P d4908192cd2f495b89bf5e33b5553a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:43:49.741402 4814 ts_tablet_manager.cc:1939] T 5e9a29e5ee5c45e38781020241d7ff0a P 0e676019fbd1414ba35874f1934bf852: Deleting consensus metadata
I20260928 06:43:49.741751 4306 catalog_manager.cc:4841] TS 0e676019fbd1414ba35874f1934bf852 (127.18.211.66:40811): tablet 5e9a29e5ee5c45e38781020241d7ff0a (table test-table [id=d111d4ef9c7c420e9db31789f62a4884]) successfully deleted
I20260928 06:43:49.742878 4812 ts_tablet_manager.cc:1918] T 5e9a29e5ee5c45e38781020241d7ff0a P d4908192cd2f495b89bf5e33b5553a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:43:49.742954 4812 log.cc:1196] T 5e9a29e5ee5c45e38781020241d7ff0a P d4908192cd2f495b89bf5e33b5553a32: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790577602522428-19277-0/minicluster-data/ts-2/wal/wals/5e9a29e5ee5c45e38781020241d7ff0a
I20260928 06:43:49.743218 4812 ts_tablet_manager.cc:1939] T 5e9a29e5ee5c45e38781020241d7ff0a P d4908192cd2f495b89bf5e33b5553a32: Deleting consensus metadata
I20260928 06:43:49.743654 4306 catalog_manager.cc:4841] TS d4908192cd2f495b89bf5e33b5553a32 (127.18.211.67:34857): tablet 5e9a29e5ee5c45e38781020241d7ff0a (table test-table [id=d111d4ef9c7c420e9db31789f62a4884]) successfully deleted
W20260928 06:43:49.744436 4308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.211.65:46451 (error 108)
I20260928 06:43:49.744714 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 4514
I20260928 06:43:49.744877 4783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:49.744761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:46451 (local address 127.0.0.1:60282)
0928 06:43:49.744808 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:49.744847 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.65:46451: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 06:43:49.747668 4308 catalog_manager.cc:4584] TS 41313ba335d24193b61537014b76dde3 (127.18.211.65:46451): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5e9a29e5ee5c45e38781020241d7ff0a: Network error: Client connection negotiation failed: client connection to 127.18.211.65:46451: connect: Connection refused (error 111)
I20260928 06:43:49.749966 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 4646
I20260928 06:43:49.753996 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 4290
2026-09-28T06:43:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (781 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:43:49 dist-test-slave-b28r krb5kdc[4818](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:43:49 dist-test-slave-b28r krb5kdc[4818](info): set up 2 sockets
Sep 28 06:43:49 dist-test-slave-b28r krb5kdc[4818](info): commencing operation
krb5kdc: starting...
W20260928 06:43:50.699729 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:43:50 dist-test-slave-b28r krb5kdc[4818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:43:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:43:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.211.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
I20260928 06:43:50.810184 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:44107
--webserver_interface=127.18.211.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:36663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:44107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:43:50.889174 4831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:43:50.889382 4831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:43:50.889412 4831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:43:50.890514 4831 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:43:50.890552 4831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:43:50.890570 4831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:43:50.890585 4831 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:43:50.890602 4831 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:43:50.892053 4831 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:36663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:44107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:44107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:43:50.892282 4831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:43:50.892477 4831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:43:50.897012 4837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:50.897030 4831 server_base.cc:1034] running on GCE node
W20260928 06:43:50.896981 4836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:50.896997 4839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:50.897400 4831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:43:50.897693 4831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:43:50.898846 4831 hybrid_clock.cc:648] HybridClock initialized: now 1790577830898834 us; error 31 us; skew 500 ppm
Sep 28 06:43:50 dist-test-slave-b28r krb5kdc[4818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577830, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:43:50.901297 4831 init.cc:377] Logged in from keytab as kudu/127.18.211.126@KRBTEST.COM (short username kudu)
I20260928 06:43:50.901731 4831 webserver.cc:460] Webserver started at http://127.18.211.126:41261/ using document root <none> and password file <none>
I20260928 06:43:50.901925 4831 fs_manager.cc:359] Metadata directory not provided
I20260928 06:43:50.901973 4831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:43:50.902096 4831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:43:50.903129 4831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "8d814e3c2ed047a3b241f27f6b45ead1"
format_stamp: "Formatted at 2026-09-28 06:43:50 on dist-test-slave-b28r"
server_key: "a907b50dd6f722d0944da5b793667c73"
server_key_iv: "3edde29ba6050301aff85bfda8e0ab82"
server_key_version: "encryptionkey@0"
I20260928 06:43:50.903481 4831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "8d814e3c2ed047a3b241f27f6b45ead1"
format_stamp: "Formatted at 2026-09-28 06:43:50 on dist-test-slave-b28r"
server_key: "a907b50dd6f722d0944da5b793667c73"
server_key_iv: "3edde29ba6050301aff85bfda8e0ab82"
server_key_version: "encryptionkey@0"
I20260928 06:43:50.904726 4831 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:43:50.905433 4846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:50.905606 4831 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:43:50.905692 4831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "8d814e3c2ed047a3b241f27f6b45ead1"
format_stamp: "Formatted at 2026-09-28 06:43:50 on dist-test-slave-b28r"
server_key: "a907b50dd6f722d0944da5b793667c73"
server_key_iv: "3edde29ba6050301aff85bfda8e0ab82"
server_key_version: "encryptionkey@0"
I20260928 06:43:50.905761 4831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:43:50.913784 4831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:43:50.915546 4831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:43:50.915668 4831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:43:50.919287 4831 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:44107
I20260928 06:43:50.919353 4898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:44107 every 8 connection(s)
I20260928 06:43:50.919623 4831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:43:50.920389 4899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:43:50.922504 4899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d814e3c2ed047a3b241f27f6b45ead1: Bootstrap starting.
I20260928 06:43:50.923009 4899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d814e3c2ed047a3b241f27f6b45ead1: Neither blocks nor log segments found. Creating new log.
I20260928 06:43:50.923280 4899 log.cc:824] T 00000000000000000000000000000000 P 8d814e3c2ed047a3b241f27f6b45ead1: Log is configured to *not* fsync() on all Append() calls
I20260928 06:43:50.923911 4899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d814e3c2ed047a3b241f27f6b45ead1: No bootstrap required, opened a new log
I20260928 06:43:50.924465 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 4831
I20260928 06:43:50.924574 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:43:50.924742 19277 external_mini_cluster.cc:1428] Setting key 832d9f27fcdd08fabe678f9db94c5659
I20260928 06:43:50.925808 4899 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d814e3c2ed047a3b241f27f6b45ead1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d814e3c2ed047a3b241f27f6b45ead1" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44107 } }
I20260928 06:43:50.925971 4899 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d814e3c2ed047a3b241f27f6b45ead1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:43:50.926008 4899 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d814e3c2ed047a3b241f27f6b45ead1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d814e3c2ed047a3b241f27f6b45ead1, State: Initialized, Role: FOLLOWER
I20260928 06:43:50.926133 4899 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d814e3c2ed047a3b241f27f6b45ead1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d814e3c2ed047a3b241f27f6b45ead1" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44107 } }
I20260928 06:43:50.926211 4899 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d814e3c2ed047a3b241f27f6b45ead1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:43:50.926242 4899 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d814e3c2ed047a3b241f27f6b45ead1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:43:50.926288 4899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d814e3c2ed047a3b241f27f6b45ead1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:43:50.927232 4899 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d814e3c2ed047a3b241f27f6b45ead1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d814e3c2ed047a3b241f27f6b45ead1" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44107 } }
I20260928 06:43:50.927322 4899 leader_election.cc:302] T 00000000000000000000000000000000 P 8d814e3c2ed047a3b241f27f6b45ead1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d814e3c2ed047a3b241f27f6b45ead1; no voters:
I20260928 06:43:50.927471 4899 leader_election.cc:288] T 00000000000000000000000000000000 P 8d814e3c2ed047a3b241f27f6b45ead1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:43:50.927512 4904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d814e3c2ed047a3b241f27f6b45ead1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:43:50.927657 4904 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d814e3c2ed047a3b241f27f6b45ead1 [term 1 LEADER]: Becoming Leader. State: Replica: 8d814e3c2ed047a3b241f27f6b45ead1, State: Running, Role: LEADER
I20260928 06:43:50.927707 4899 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d814e3c2ed047a3b241f27f6b45ead1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:43:50.927814 4904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d814e3c2ed047a3b241f27f6b45ead1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d814e3c2ed047a3b241f27f6b45ead1" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44107 } }
I20260928 06:43:50.928366 4905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d814e3c2ed047a3b241f27f6b45ead1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d814e3c2ed047a3b241f27f6b45ead1. Latest consensus state: current_term: 1 leader_uuid: "8d814e3c2ed047a3b241f27f6b45ead1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d814e3c2ed047a3b241f27f6b45ead1" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44107 } } }
I20260928 06:43:50.928454 4905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d814e3c2ed047a3b241f27f6b45ead1 [sys.catalog]: This master's current role is: LEADER
I20260928 06:43:50.928694 4906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d814e3c2ed047a3b241f27f6b45ead1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d814e3c2ed047a3b241f27f6b45ead1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d814e3c2ed047a3b241f27f6b45ead1" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44107 } } }
I20260928 06:43:50.928998 4906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d814e3c2ed047a3b241f27f6b45ead1 [sys.catalog]: This master's current role is: LEADER
I20260928 06:43:50.929152 4913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:43:50.929674 4913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:43:50.931209 4913 catalog_manager.cc:1347] Generated new cluster ID: bf05761766cb4942a43f2b362551a42b
I20260928 06:43:50.931263 4913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:43:50 dist-test-slave-b28r krb5kdc[4818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:43:50.935722 4903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:50.925191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48510 (local address 127.18.211.126:44107)
0928 06:43:50.925430 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:50.925438 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:50.925463 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:43:50.925763 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:50.925768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:50.925811 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:50.925917 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:50.926700 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:50.928038 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:50.929138 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:50.929686 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:50.929795 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:50.932549 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:50.932553 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:50.932554 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:43:50.932565 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:50.934029 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:50.934344 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:50.934347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:50.934347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:50.934447 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:50.934621 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:50.934622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:50.934622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:50.934845 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:50.935420 (+ 575us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:50.935478 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 06:43:50.935577 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":139,"threads_started":1}
I20260928 06:43:50.938776 4913 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:43:50.939347 4913 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:43:50.943178 4913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d814e3c2ed047a3b241f27f6b45ead1: Generated new TSK 0
I20260928 06:43:50.943338 4913 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.211.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab.
Entry for principal kudu/127.18.211.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab.
I20260928 06:43:50.969851 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.65:0
--local_ip_for_outbound_sockets=127.18.211.65
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:44107
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:36663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:43:51.049212 4925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:43:51.049419 4925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:43:51.049450 4925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:43:51.050588 4925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:43:51.050627 4925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:43:51.050670 4925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.65
I20260928 06:43:51.052145 4925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:36663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:44107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.65
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:43:51.052366 4925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:43:51.052588 4925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:43:51.057247 4930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:51.057211 4931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:51.057189 4933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:51.057520 4925 server_base.cc:1034] running on GCE node
I20260928 06:43:51.057737 4925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:43:51.057986 4925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:43:51.059141 4925 hybrid_clock.cc:648] HybridClock initialized: now 1790577831059115 us; error 41 us; skew 500 ppm
Sep 28 06:43:51 dist-test-slave-b28r krb5kdc[4818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577831, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:43:51.061661 4925 init.cc:377] Logged in from keytab as kudu/127.18.211.65@KRBTEST.COM (short username kudu)
I20260928 06:43:51.062090 4925 webserver.cc:460] Webserver started at http://127.18.211.65:34901/ using document root <none> and password file <none>
I20260928 06:43:51.062300 4925 fs_manager.cc:359] Metadata directory not provided
I20260928 06:43:51.062351 4925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:43:51.062471 4925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:43:51.063557 4925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-0/data/instance:
uuid: "69c655706f674bd790685ede15052973"
format_stamp: "Formatted at 2026-09-28 06:43:51 on dist-test-slave-b28r"
server_key: "a8097951743f4fa70fb559987d7936bb"
server_key_iv: "791c759815624c32914d10d4383d87ce"
server_key_version: "encryptionkey@0"
I20260928 06:43:51.063890 4925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance:
uuid: "69c655706f674bd790685ede15052973"
format_stamp: "Formatted at 2026-09-28 06:43:51 on dist-test-slave-b28r"
server_key: "a8097951743f4fa70fb559987d7936bb"
server_key_iv: "791c759815624c32914d10d4383d87ce"
server_key_version: "encryptionkey@0"
I20260928 06:43:51.065104 4925 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:43:51.065752 4941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:51.065917 4925 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:43:51.065997 4925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-0/wal
uuid: "69c655706f674bd790685ede15052973"
format_stamp: "Formatted at 2026-09-28 06:43:51 on dist-test-slave-b28r"
server_key: "a8097951743f4fa70fb559987d7936bb"
server_key_iv: "791c759815624c32914d10d4383d87ce"
server_key_version: "encryptionkey@0"
I20260928 06:43:51.066056 4925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:43:51.084563 4925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:43:51.086397 4925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:43:51.086539 4925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:43:51.086777 4925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:43:51.087074 4925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:43:51.087139 4925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:51.087170 4925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:43:51.087194 4925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:51.092589 4925 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.65:45013
I20260928 06:43:51.092648 5056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.65:45013 every 8 connection(s)
I20260928 06:43:51.092937 4925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
I20260928 06:43:51.094707 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 4925
I20260928 06:43:51.094838 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance
I20260928 06:43:51.095024 19277 external_mini_cluster.cc:1428] Setting key 8223537b5e15658d259f73b257531c91
Sep 28 06:43:51 dist-test-slave-b28r krb5kdc[4818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577831, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:43:51.102655 4903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:51.093640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:45495 (local address 127.18.211.126:44107)
0928 06:43:51.093695 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:51.093697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:51.093888 (+ 191us) server_negotiation.cc:407] Connection header received
0928 06:43:51.094440 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:51.094441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:51.094454 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:51.094506 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:51.095595 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.096156 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.097149 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.097371 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.097428 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:51.099714 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:51.099716 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:51.099716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:43:51.099729 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:51.101281 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:51.101690 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:51.101691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:51.101691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:51.101754 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:51.101969 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:51.101969 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:51.101970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:51.102125 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:51.102196 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:51.102455 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 06:43:51.102557 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 06:43:51.102705 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:51.093602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44107 (local address 127.18.211.65:45495)
0928 06:43:51.093766 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:51.093807 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:51.094336 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:51.094597 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:51.094601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:51.095285 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:43:51.095499 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:51.095504 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.096250 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.096251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:51.097061 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:51.097063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.097443 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.097444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:51.097573 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:51.098056 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:43:51.098065 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:43:51.099615 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:43:51.101382 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:43:51.101390 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:43:51.101391 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:43:51.101615 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:43:51.101823 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:43:51.101823 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:43:51.101823 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:43:51.101914 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:43:51.102191 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:43:51.102197 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:43:51.102327 (+ 130us) client_negotiation.cc:769] Sending connection context
0928 06:43:51.102415 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 06:43:51.102537 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":84,"threads_started":1}
I20260928 06:43:51.103236 5057 heartbeater.cc:347] Connected to a master server at 127.18.211.126:44107
WARNING: no policy specified for kudu/127.18.211.66@KRBTEST.COM; defaulting to no policy
I20260928 06:43:51.103332 5057 heartbeater.cc:464] Registering TS with master...
I20260928 06:43:51.103503 5057 heartbeater.cc:511] Master 127.18.211.126:44107 requested a full tablet report, sending...
I20260928 06:43:51.104014 4863 ts_manager.cc:194] Registered new tserver with Master: 69c655706f674bd790685ede15052973 (127.18.211.65:45013)
I20260928 06:43:51.104640 4863 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.65@KRBTEST.COM'} at 127.18.211.65:45495
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab.
Entry for principal kudu/127.18.211.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab.
I20260928 06:43:51.119314 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.66:0
--local_ip_for_outbound_sockets=127.18.211.66
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:44107
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:36663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:43:51.199295 5062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:43:51.199497 5062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:43:51.199519 5062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:43:51.200673 5062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:43:51.200716 5062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:43:51.200742 5062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.66
I20260928 06:43:51.202181 5062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:36663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:44107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.66
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:43:51.202402 5062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:43:51.202620 5062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:43:51.207428 5067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:51.207348 5070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:51.207515 5068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:51.207526 5062 server_base.cc:1034] running on GCE node
I20260928 06:43:51.207818 5062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:43:51.208050 5062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:43:51.209204 5062 hybrid_clock.cc:648] HybridClock initialized: now 1790577831209186 us; error 30 us; skew 500 ppm
Sep 28 06:43:51 dist-test-slave-b28r krb5kdc[4818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577831, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:43:51.211773 5062 init.cc:377] Logged in from keytab as kudu/127.18.211.66@KRBTEST.COM (short username kudu)
I20260928 06:43:51.212188 5062 webserver.cc:460] Webserver started at http://127.18.211.66:42289/ using document root <none> and password file <none>
I20260928 06:43:51.212404 5062 fs_manager.cc:359] Metadata directory not provided
I20260928 06:43:51.212452 5062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:43:51.212579 5062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:43:51.213645 5062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-1/data/instance:
uuid: "04268ae26e9c44b18e0b5c2c095bb2c3"
format_stamp: "Formatted at 2026-09-28 06:43:51 on dist-test-slave-b28r"
server_key: "999284425ddaaf10c16e086de4a05830"
server_key_iv: "e5bb9e77b50116290c40080f471e4c6a"
server_key_version: "encryptionkey@0"
I20260928 06:43:51.213971 5062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance:
uuid: "04268ae26e9c44b18e0b5c2c095bb2c3"
format_stamp: "Formatted at 2026-09-28 06:43:51 on dist-test-slave-b28r"
server_key: "999284425ddaaf10c16e086de4a05830"
server_key_iv: "e5bb9e77b50116290c40080f471e4c6a"
server_key_version: "encryptionkey@0"
I20260928 06:43:51.215226 5062 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:43:51.216006 5077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:51.216159 5062 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:43:51.216279 5062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-1/wal
uuid: "04268ae26e9c44b18e0b5c2c095bb2c3"
format_stamp: "Formatted at 2026-09-28 06:43:51 on dist-test-slave-b28r"
server_key: "999284425ddaaf10c16e086de4a05830"
server_key_iv: "e5bb9e77b50116290c40080f471e4c6a"
server_key_version: "encryptionkey@0"
I20260928 06:43:51.216343 5062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:43:51.240057 5062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:43:51.242471 5062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:43:51.242645 5062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:43:51.242908 5062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:43:51.243268 5062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:43:51.243307 5062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:51.243342 5062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:43:51.243364 5062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:51.249603 5062 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.66:33005
I20260928 06:43:51.249660 5191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.66:33005 every 8 connection(s)
I20260928 06:43:51.249991 5062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
I20260928 06:43:51.254155 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 5062
I20260928 06:43:51.254307 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance
I20260928 06:43:51.254484 19277 external_mini_cluster.cc:1428] Setting key b3b8ae6877f0853aeb442247ce8a721a
Sep 28 06:43:51 dist-test-slave-b28r krb5kdc[4818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577831, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:43:51.259660 4903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:51.250758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:60567 (local address 127.18.211.126:44107)
0928 06:43:51.250830 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:51.250832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:51.251035 (+ 203us) server_negotiation.cc:407] Connection header received
0928 06:43:51.251617 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:51.251618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:51.251629 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:51.251700 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:51.252679 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.253197 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.254235 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.254487 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.254542 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:51.256874 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:51.256875 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:51.256875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:43:51.256881 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:51.258318 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:51.258713 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:51.258714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:51.258714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:51.258782 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:51.259010 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:51.259011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:51.259011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:51.259160 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:51.259218 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:51.259485 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 06:43:51.259572 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:43:51.259837 5194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:51.250699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44107 (local address 127.18.211.66:60567)
0928 06:43:51.250898 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:51.250941 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:51.251502 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:51.251769 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:51.251773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:51.252388 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:43:51.252585 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:51.252590 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.253311 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.253312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:51.254138 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:51.254141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.254573 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.254574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:51.254729 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:51.255324 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:43:51.255332 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:43:51.256770 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:43:51.258401 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:43:51.258409 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:43:51.258410 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:43:51.258623 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:43:51.258842 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:43:51.258842 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:43:51.258843 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:43:51.258953 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:43:51.259216 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:43:51.259223 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:43:51.259361 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 06:43:51.259456 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 06:43:51.259610 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":111,"threads_started":1}
I20260928 06:43:51.260251 5192 heartbeater.cc:347] Connected to a master server at 127.18.211.126:44107
I20260928 06:43:51.260342 5192 heartbeater.cc:464] Registering TS with master...
I20260928 06:43:51.260506 5192 heartbeater.cc:511] Master 127.18.211.126:44107 requested a full tablet report, sending...
I20260928 06:43:51.260876 4863 ts_manager.cc:194] Registered new tserver with Master: 04268ae26e9c44b18e0b5c2c095bb2c3 (127.18.211.66:33005)
I20260928 06:43:51.261385 4863 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.66@KRBTEST.COM'} at 127.18.211.66:60567
WARNING: no policy specified for kudu/127.18.211.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab.
Entry for principal kudu/127.18.211.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab.
I20260928 06:43:51.277375 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.67:0
--local_ip_for_outbound_sockets=127.18.211.67
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:44107
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:36663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:43:51.358156 5197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:43:51.358361 5197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:43:51.358384 5197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:43:51.359541 5197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:43:51.359586 5197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:43:51.359613 5197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.67
I20260928 06:43:51.361058 5197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:36663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:44107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.67
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:43:51.361269 5197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:43:51.361486 5197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:43:51.366246 5202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:51.366271 5205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:51.366278 5203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:51.366334 5197 server_base.cc:1034] running on GCE node
I20260928 06:43:51.366624 5197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:43:51.366837 5197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:43:51.367972 5197 hybrid_clock.cc:648] HybridClock initialized: now 1790577831367962 us; error 33 us; skew 500 ppm
Sep 28 06:43:51 dist-test-slave-b28r krb5kdc[4818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577831, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:43:51.370298 5197 init.cc:377] Logged in from keytab as kudu/127.18.211.67@KRBTEST.COM (short username kudu)
I20260928 06:43:51.370682 5197 webserver.cc:460] Webserver started at http://127.18.211.67:35647/ using document root <none> and password file <none>
I20260928 06:43:51.370872 5197 fs_manager.cc:359] Metadata directory not provided
I20260928 06:43:51.370913 5197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:43:51.371017 5197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:43:51.372014 5197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-2/data/instance:
uuid: "d86f1e60244c42579e5f90793e1b7ed6"
format_stamp: "Formatted at 2026-09-28 06:43:51 on dist-test-slave-b28r"
server_key: "40eb2178920d0c738fda710624c2b59c"
server_key_iv: "5ed045ac3dee657f487f21e3bcba5146"
server_key_version: "encryptionkey@0"
I20260928 06:43:51.372318 5197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance:
uuid: "d86f1e60244c42579e5f90793e1b7ed6"
format_stamp: "Formatted at 2026-09-28 06:43:51 on dist-test-slave-b28r"
server_key: "40eb2178920d0c738fda710624c2b59c"
server_key_iv: "5ed045ac3dee657f487f21e3bcba5146"
server_key_version: "encryptionkey@0"
I20260928 06:43:51.373577 5197 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:43:51.374260 5212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:51.374406 5197 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:43:51.374475 5197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-2/wal
uuid: "d86f1e60244c42579e5f90793e1b7ed6"
format_stamp: "Formatted at 2026-09-28 06:43:51 on dist-test-slave-b28r"
server_key: "40eb2178920d0c738fda710624c2b59c"
server_key_iv: "5ed045ac3dee657f487f21e3bcba5146"
server_key_version: "encryptionkey@0"
I20260928 06:43:51.374542 5197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:43:51.419852 5197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:43:51.421782 5197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:43:51.421905 5197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:43:51.422137 5197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:43:51.422437 5197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:43:51.422472 5197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:51.422493 5197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:43:51.422516 5197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:51.428737 5197 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.67:41913
I20260928 06:43:51.428778 5326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.67:41913 every 8 connection(s)
I20260928 06:43:51.429109 5197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
I20260928 06:43:51.432015 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 5197
I20260928 06:43:51.432134 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance
I20260928 06:43:51.432363 19277 external_mini_cluster.cc:1428] Setting key 6ac10b52b8272659a5f05b2c0ee89fb6
Sep 28 06:43:51 dist-test-slave-b28r krb5kdc[4818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577831, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:43:51.438512 4903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:51.429794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:58593 (local address 127.18.211.126:44107)
0928 06:43:51.429853 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:51.429855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:51.430041 (+ 186us) server_negotiation.cc:407] Connection header received
0928 06:43:51.430501 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:51.430502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:51.430512 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:51.430560 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:51.431506 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.432020 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.433398 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.433586 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.433634 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:51.435864 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:51.435865 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:51.435866 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:43:51.435872 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:51.437099 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:51.437568 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:51.437569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:51.437569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:51.437631 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:51.437911 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:51.437912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:51.437913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:51.438025 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:51.438077 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:51.438380 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 06:43:51.438448 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 06:43:51.438673 5329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:51.429784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44107 (local address 127.18.211.67:58593)
0928 06:43:51.429928 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:51.429962 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:51.430421 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:51.430621 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:51.430625 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:51.431237 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:43:51.431429 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:51.431434 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.432127 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.432129 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:51.433319 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:51.433322 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.433675 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.433676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:51.433800 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:51.434268 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:43:51.434277 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:43:51.435718 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:43:51.437185 (+ 1467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:43:51.437192 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:43:51.437194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:43:51.437460 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:43:51.437717 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:43:51.437718 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:43:51.437719 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:43:51.437842 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:43:51.438082 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:43:51.438087 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:43:51.438244 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 06:43:51.438342 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 06:43:51.438469 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260928 06:43:51.439165 5327 heartbeater.cc:347] Connected to a master server at 127.18.211.126:44107
I20260928 06:43:51.439265 5327 heartbeater.cc:464] Registering TS with master...
I20260928 06:43:51.439409 5327 heartbeater.cc:511] Master 127.18.211.126:44107 requested a full tablet report, sending...
I20260928 06:43:51.439764 4863 ts_manager.cc:194] Registered new tserver with Master: d86f1e60244c42579e5f90793e1b7ed6 (127.18.211.67:41913)
I20260928 06:43:51.440104 4863 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.67@KRBTEST.COM'} at 127.18.211.67:58593
I20260928 06:43:51.443933 19277 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:43:51.450343 4903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:51.445092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48516 (local address 127.18.211.126:44107)
0928 06:43:51.445150 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:51.445151 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:51.445207 (+ 56us) server_negotiation.cc:407] Connection header received
0928 06:43:51.445255 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:51.445256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:51.445269 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:51.445314 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:51.445795 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.446256 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.446961 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.447192 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.447236 (+ 44us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:51.448343 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:51.448345 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:51.448345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:43:51.448351 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:51.449475 (+ 1124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:51.449788 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:51.449789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:51.449789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:51.449835 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:51.449956 (+ 121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:51.449956 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:51.449956 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:51.450063 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:51.450134 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:51.450211 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 06:43:51.450256 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 06:43:51.451743 4863 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48516:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:43:51.453243 4863 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:43:51.459295 5342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:51.455285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:33005 (local address 127.0.0.1:59838)
0928 06:43:51.455587 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:51.455597 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:51.455649 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:51.455991 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:51.455994 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:51.455996 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:43:51.456077 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:51.456080 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.457073 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.457074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:51.458298 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:51.458301 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.458986 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.458987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:51.459063 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:51.459076 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 06:43:51.459166 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 06:43:51.459220 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":75,"threads_started":1}
I20260928 06:43:51.459307 5340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:51.455384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59838 (local address 127.18.211.66:33005)
0928 06:43:51.455793 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:51.455799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:51.455809 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:43:51.455870 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:51.455872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:51.455887 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:51.455957 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:43:51.456165 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.456977 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.458380 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.458913 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.458976 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:51.459036 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:51.459184 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 06:43:51.459228 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":83,"threads_started":1}
I20260928 06:43:51.459604 5338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:51.455482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:45013 (local address 127.0.0.1:42746)
0928 06:43:51.455627 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:51.455633 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:51.455665 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:51.456034 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:51.456035 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:51.456038 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:43:51.456085 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:51.456087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.457377 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.457378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:51.458459 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:51.458462 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.459372 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.459373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:51.459439 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:51.459447 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 06:43:51.459475 (+ 28us) client_negotiation.cc:240] Negotiation successful
0928 06:43:51.459513 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":37,"thread_start_us":67,"threads_started":1}
I20260928 06:43:51.459604 5339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:51.455281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:41913 (local address 127.0.0.1:42224)
0928 06:43:51.455448 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:51.455488 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:51.455549 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:51.456312 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:51.456313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:51.456314 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:43:51.456348 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:51.456350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.457550 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.457551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:51.458422 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:51.458424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.459370 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.459372 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:51.459439 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:51.459445 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 06:43:51.459475 (+ 30us) client_negotiation.cc:240] Negotiation successful
0928 06:43:51.459513 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":61,"threads_started":1}
I20260928 06:43:51.459942 5341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:51.455389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42224 (local address 127.18.211.67:41913)
0928 06:43:51.456153 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:51.456157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:51.456166 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:43:51.456200 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:51.456202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:51.456217 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:51.456414 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:43:51.456472 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.457468 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.458503 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.459268 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.459689 (+ 421us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:51.459789 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:51.459816 (+ 27us) server_negotiation.cc:299] Negotiation successful
0928 06:43:51.459860 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":82,"threads_started":1}
I20260928 06:43:51.460018 5343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:51.455711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42746 (local address 127.18.211.65:45013)
0928 06:43:51.455878 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:51.455883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:51.455892 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:43:51.455925 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:51.455926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:51.455939 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:51.456021 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:43:51.456171 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.457280 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.458541 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.459280 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.459820 (+ 540us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:51.459878 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:51.459902 (+ 24us) server_negotiation.cc:299] Negotiation successful
0928 06:43:51.459944 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":93,"threads_started":1}
I20260928 06:43:51.460052 5126 tablet_service.cc:1467] Processing CreateTablet for tablet 5024546682c2420db8540c03ee0d3cdc (DEFAULT_TABLE table=test-table [id=a93252b5cf474cb1ac2de62680655881]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:43:51.460436 5126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5024546682c2420db8540c03ee0d3cdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:43:51.460573 5261 tablet_service.cc:1467] Processing CreateTablet for tablet 5024546682c2420db8540c03ee0d3cdc (DEFAULT_TABLE table=test-table [id=a93252b5cf474cb1ac2de62680655881]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:43:51.460600 4990 tablet_service.cc:1467] Processing CreateTablet for tablet 5024546682c2420db8540c03ee0d3cdc (DEFAULT_TABLE table=test-table [id=a93252b5cf474cb1ac2de62680655881]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:43:51.460911 5261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5024546682c2420db8540c03ee0d3cdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:43:51.460911 4990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5024546682c2420db8540c03ee0d3cdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:43:51.462982 5344 tablet_bootstrap.cc:492] T 5024546682c2420db8540c03ee0d3cdc P 04268ae26e9c44b18e0b5c2c095bb2c3: Bootstrap starting.
I20260928 06:43:51.463671 5344 tablet_bootstrap.cc:654] T 5024546682c2420db8540c03ee0d3cdc P 04268ae26e9c44b18e0b5c2c095bb2c3: Neither blocks nor log segments found. Creating new log.
I20260928 06:43:51.464062 5345 tablet_bootstrap.cc:492] T 5024546682c2420db8540c03ee0d3cdc P 69c655706f674bd790685ede15052973: Bootstrap starting.
I20260928 06:43:51.464227 5344 log.cc:824] T 5024546682c2420db8540c03ee0d3cdc P 04268ae26e9c44b18e0b5c2c095bb2c3: Log is configured to *not* fsync() on all Append() calls
I20260928 06:43:51.464656 5345 tablet_bootstrap.cc:654] T 5024546682c2420db8540c03ee0d3cdc P 69c655706f674bd790685ede15052973: Neither blocks nor log segments found. Creating new log.
I20260928 06:43:51.464986 5344 tablet_bootstrap.cc:492] T 5024546682c2420db8540c03ee0d3cdc P 04268ae26e9c44b18e0b5c2c095bb2c3: No bootstrap required, opened a new log
I20260928 06:43:51.465057 5344 ts_tablet_manager.cc:1397] T 5024546682c2420db8540c03ee0d3cdc P 04268ae26e9c44b18e0b5c2c095bb2c3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:43:51.465243 5345 log.cc:824] T 5024546682c2420db8540c03ee0d3cdc P 69c655706f674bd790685ede15052973: Log is configured to *not* fsync() on all Append() calls
I20260928 06:43:51.465574 5346 tablet_bootstrap.cc:492] T 5024546682c2420db8540c03ee0d3cdc P d86f1e60244c42579e5f90793e1b7ed6: Bootstrap starting.
I20260928 06:43:51.465910 5345 tablet_bootstrap.cc:492] T 5024546682c2420db8540c03ee0d3cdc P 69c655706f674bd790685ede15052973: No bootstrap required, opened a new log
I20260928 06:43:51.465976 5345 ts_tablet_manager.cc:1397] T 5024546682c2420db8540c03ee0d3cdc P 69c655706f674bd790685ede15052973: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:43:51.466368 5346 tablet_bootstrap.cc:654] T 5024546682c2420db8540c03ee0d3cdc P d86f1e60244c42579e5f90793e1b7ed6: Neither blocks nor log segments found. Creating new log.
I20260928 06:43:51.466949 5346 log.cc:824] T 5024546682c2420db8540c03ee0d3cdc P d86f1e60244c42579e5f90793e1b7ed6: Log is configured to *not* fsync() on all Append() calls
I20260928 06:43:51.466982 5344 raft_consensus.cc:348] T 5024546682c2420db8540c03ee0d3cdc P 04268ae26e9c44b18e0b5c2c095bb2c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d86f1e60244c42579e5f90793e1b7ed6" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 41913 } } peers { permanent_uuid: "04268ae26e9c44b18e0b5c2c095bb2c3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33005 } } peers { permanent_uuid: "69c655706f674bd790685ede15052973" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45013 } }
I20260928 06:43:51.467195 5344 raft_consensus.cc:374] T 5024546682c2420db8540c03ee0d3cdc P 04268ae26e9c44b18e0b5c2c095bb2c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:43:51.467244 5344 raft_consensus.cc:729] T 5024546682c2420db8540c03ee0d3cdc P 04268ae26e9c44b18e0b5c2c095bb2c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04268ae26e9c44b18e0b5c2c095bb2c3, State: Initialized, Role: FOLLOWER
I20260928 06:43:51.467370 5344 consensus_queue.cc:260] T 5024546682c2420db8540c03ee0d3cdc P 04268ae26e9c44b18e0b5c2c095bb2c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d86f1e60244c42579e5f90793e1b7ed6" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 41913 } } peers { permanent_uuid: "04268ae26e9c44b18e0b5c2c095bb2c3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33005 } } peers { permanent_uuid: "69c655706f674bd790685ede15052973" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45013 } }
I20260928 06:43:51.467648 5192 heartbeater.cc:503] Master 127.18.211.126:44107 was elected leader, sending a full tablet report...
I20260928 06:43:51.467605 5345 raft_consensus.cc:348] T 5024546682c2420db8540c03ee0d3cdc P 69c655706f674bd790685ede15052973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d86f1e60244c42579e5f90793e1b7ed6" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 41913 } } peers { permanent_uuid: "04268ae26e9c44b18e0b5c2c095bb2c3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33005 } } peers { permanent_uuid: "69c655706f674bd790685ede15052973" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45013 } }
I20260928 06:43:51.467772 5345 raft_consensus.cc:374] T 5024546682c2420db8540c03ee0d3cdc P 69c655706f674bd790685ede15052973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:43:51.467811 5345 raft_consensus.cc:729] T 5024546682c2420db8540c03ee0d3cdc P 69c655706f674bd790685ede15052973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69c655706f674bd790685ede15052973, State: Initialized, Role: FOLLOWER
I20260928 06:43:51.467865 5346 tablet_bootstrap.cc:492] T 5024546682c2420db8540c03ee0d3cdc P d86f1e60244c42579e5f90793e1b7ed6: No bootstrap required, opened a new log
I20260928 06:43:51.467931 5345 consensus_queue.cc:260] T 5024546682c2420db8540c03ee0d3cdc P 69c655706f674bd790685ede15052973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d86f1e60244c42579e5f90793e1b7ed6" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 41913 } } peers { permanent_uuid: "04268ae26e9c44b18e0b5c2c095bb2c3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33005 } } peers { permanent_uuid: "69c655706f674bd790685ede15052973" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45013 } }
I20260928 06:43:51.467942 5346 ts_tablet_manager.cc:1397] T 5024546682c2420db8540c03ee0d3cdc P d86f1e60244c42579e5f90793e1b7ed6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 06:43:51.467636 5344 ts_tablet_manager.cc:1428] T 5024546682c2420db8540c03ee0d3cdc P 04268ae26e9c44b18e0b5c2c095bb2c3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:43:51.468191 5345 ts_tablet_manager.cc:1428] T 5024546682c2420db8540c03ee0d3cdc P 69c655706f674bd790685ede15052973: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:43:51.468225 5057 heartbeater.cc:503] Master 127.18.211.126:44107 was elected leader, sending a full tablet report...
I20260928 06:43:51.469637 5346 raft_consensus.cc:348] T 5024546682c2420db8540c03ee0d3cdc P d86f1e60244c42579e5f90793e1b7ed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d86f1e60244c42579e5f90793e1b7ed6" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 41913 } } peers { permanent_uuid: "04268ae26e9c44b18e0b5c2c095bb2c3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33005 } } peers { permanent_uuid: "69c655706f674bd790685ede15052973" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45013 } }
I20260928 06:43:51.469760 5346 raft_consensus.cc:374] T 5024546682c2420db8540c03ee0d3cdc P d86f1e60244c42579e5f90793e1b7ed6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:43:51.469780 5346 raft_consensus.cc:729] T 5024546682c2420db8540c03ee0d3cdc P d86f1e60244c42579e5f90793e1b7ed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d86f1e60244c42579e5f90793e1b7ed6, State: Initialized, Role: FOLLOWER
I20260928 06:43:51.469858 5346 consensus_queue.cc:260] T 5024546682c2420db8540c03ee0d3cdc P d86f1e60244c42579e5f90793e1b7ed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d86f1e60244c42579e5f90793e1b7ed6" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 41913 } } peers { permanent_uuid: "04268ae26e9c44b18e0b5c2c095bb2c3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33005 } } peers { permanent_uuid: "69c655706f674bd790685ede15052973" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45013 } }
I20260928 06:43:51.470044 5346 ts_tablet_manager.cc:1428] T 5024546682c2420db8540c03ee0d3cdc P d86f1e60244c42579e5f90793e1b7ed6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:43:51.470079 5327 heartbeater.cc:503] Master 127.18.211.126:44107 was elected leader, sending a full tablet report...
I20260928 06:43:51.486336 5351 raft_consensus.cc:482] T 5024546682c2420db8540c03ee0d3cdc P 69c655706f674bd790685ede15052973 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:43:51.486440 5351 raft_consensus.cc:504] T 5024546682c2420db8540c03ee0d3cdc P 69c655706f674bd790685ede15052973 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d86f1e60244c42579e5f90793e1b7ed6" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 41913 } } peers { permanent_uuid: "04268ae26e9c44b18e0b5c2c095bb2c3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33005 } } peers { permanent_uuid: "69c655706f674bd790685ede15052973" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45013 } }
I20260928 06:43:51.486747 5351 leader_election.cc:288] T 5024546682c2420db8540c03ee0d3cdc P 69c655706f674bd790685ede15052973 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d86f1e60244c42579e5f90793e1b7ed6 (127.18.211.67:41913), 04268ae26e9c44b18e0b5c2c095bb2c3 (127.18.211.66:33005)
I20260928 06:43:51.489436 5341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:51.486957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:53441 (local address 127.18.211.67:41913)
0928 06:43:51.487009 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:51.487010 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:51.487019 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:43:51.487105 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:51.487107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:51.487121 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:51.487180 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:43:51.487328 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.487831 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.488741 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.489127 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.489188 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:51.489195 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:51.489332 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 06:43:51.489372 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 06:43:51.489470 5340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:51.487014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:57629 (local address 127.18.211.66:33005)
0928 06:43:51.487071 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:51.487073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:51.487163 (+ 90us) server_negotiation.cc:407] Connection header received
0928 06:43:51.487211 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:51.487212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:51.487222 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:51.487280 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:43:51.487412 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.487889 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.488804 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.489171 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.489241 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:51.489249 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:51.489377 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 06:43:51.489415 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 06:43:51.489454 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:51.486893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:41913 (local address 127.18.211.65:53441)
0928 06:43:51.486949 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:51.486963 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:51.487009 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:51.487195 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:51.487196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:51.487198 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:43:51.487269 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:51.487270 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.487907 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.487907 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:51.488663 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:51.488666 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.489193 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.489194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:51.489272 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:51.489275 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:43:51.489319 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 06:43:51.489355 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":17}
I20260928 06:43:51.489552 5353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:51.486893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:33005 (local address 127.18.211.65:57629)
0928 06:43:51.487064 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:51.487075 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:51.487162 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:51.487284 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:51.487284 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:51.487285 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:43:51.487355 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:51.487357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.487962 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.487962 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:51.488718 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:51.488721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.489249 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.489250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:51.489305 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:51.489309 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:43:51.489359 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 06:43:51.489394 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":11,"thread_start_us":101,"threads_started":1}
I20260928 06:43:51.489863 5281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5024546682c2420db8540c03ee0d3cdc" candidate_uuid: "69c655706f674bd790685ede15052973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d86f1e60244c42579e5f90793e1b7ed6" is_pre_election: true
I20260928 06:43:51.489930 5146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5024546682c2420db8540c03ee0d3cdc" candidate_uuid: "69c655706f674bd790685ede15052973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04268ae26e9c44b18e0b5c2c095bb2c3" is_pre_election: true
I20260928 06:43:51.489989 5281 raft_consensus.cc:2454] T 5024546682c2420db8540c03ee0d3cdc P d86f1e60244c42579e5f90793e1b7ed6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69c655706f674bd790685ede15052973 in term 0.
I20260928 06:43:51.490075 5146 raft_consensus.cc:2454] T 5024546682c2420db8540c03ee0d3cdc P 04268ae26e9c44b18e0b5c2c095bb2c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69c655706f674bd790685ede15052973 in term 0.
I20260928 06:43:51.490195 4944 leader_election.cc:302] T 5024546682c2420db8540c03ee0d3cdc P 69c655706f674bd790685ede15052973 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69c655706f674bd790685ede15052973, d86f1e60244c42579e5f90793e1b7ed6; no voters:
I20260928 06:43:51.490335 5351 raft_consensus.cc:2781] T 5024546682c2420db8540c03ee0d3cdc P 69c655706f674bd790685ede15052973 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:43:51.490391 5351 raft_consensus.cc:482] T 5024546682c2420db8540c03ee0d3cdc P 69c655706f674bd790685ede15052973 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:43:51.490406 5351 raft_consensus.cc:3037] T 5024546682c2420db8540c03ee0d3cdc P 69c655706f674bd790685ede15052973 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:43:51.491071 5351 raft_consensus.cc:504] T 5024546682c2420db8540c03ee0d3cdc P 69c655706f674bd790685ede15052973 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d86f1e60244c42579e5f90793e1b7ed6" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 41913 } } peers { permanent_uuid: "04268ae26e9c44b18e0b5c2c095bb2c3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33005 } } peers { permanent_uuid: "69c655706f674bd790685ede15052973" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45013 } }
I20260928 06:43:51.491240 5351 leader_election.cc:288] T 5024546682c2420db8540c03ee0d3cdc P 69c655706f674bd790685ede15052973 [CANDIDATE]: Term 1 election: Requested vote from peers d86f1e60244c42579e5f90793e1b7ed6 (127.18.211.67:41913), 04268ae26e9c44b18e0b5c2c095bb2c3 (127.18.211.66:33005)
I20260928 06:43:51.491379 5281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5024546682c2420db8540c03ee0d3cdc" candidate_uuid: "69c655706f674bd790685ede15052973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d86f1e60244c42579e5f90793e1b7ed6"
I20260928 06:43:51.491376 5146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5024546682c2420db8540c03ee0d3cdc" candidate_uuid: "69c655706f674bd790685ede15052973" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04268ae26e9c44b18e0b5c2c095bb2c3"
I20260928 06:43:51.491446 5281 raft_consensus.cc:3037] T 5024546682c2420db8540c03ee0d3cdc P d86f1e60244c42579e5f90793e1b7ed6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:43:51.491447 5146 raft_consensus.cc:3037] T 5024546682c2420db8540c03ee0d3cdc P 04268ae26e9c44b18e0b5c2c095bb2c3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:43:51.492266 5281 raft_consensus.cc:2454] T 5024546682c2420db8540c03ee0d3cdc P d86f1e60244c42579e5f90793e1b7ed6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69c655706f674bd790685ede15052973 in term 1.
I20260928 06:43:51.492266 5146 raft_consensus.cc:2454] T 5024546682c2420db8540c03ee0d3cdc P 04268ae26e9c44b18e0b5c2c095bb2c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69c655706f674bd790685ede15052973 in term 1.
I20260928 06:43:51.492435 4945 leader_election.cc:302] T 5024546682c2420db8540c03ee0d3cdc P 69c655706f674bd790685ede15052973 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04268ae26e9c44b18e0b5c2c095bb2c3, 69c655706f674bd790685ede15052973; no voters:
I20260928 06:43:51.492524 5351 raft_consensus.cc:2781] T 5024546682c2420db8540c03ee0d3cdc P 69c655706f674bd790685ede15052973 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:43:51.492662 5351 raft_consensus.cc:686] T 5024546682c2420db8540c03ee0d3cdc P 69c655706f674bd790685ede15052973 [term 1 LEADER]: Becoming Leader. State: Replica: 69c655706f674bd790685ede15052973, State: Running, Role: LEADER
I20260928 06:43:51.492733 5351 consensus_queue.cc:237] T 5024546682c2420db8540c03ee0d3cdc P 69c655706f674bd790685ede15052973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d86f1e60244c42579e5f90793e1b7ed6" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 41913 } } peers { permanent_uuid: "04268ae26e9c44b18e0b5c2c095bb2c3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33005 } } peers { permanent_uuid: "69c655706f674bd790685ede15052973" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45013 } }
I20260928 06:43:51.493469 4863 catalog_manager.cc:5495] T 5024546682c2420db8540c03ee0d3cdc P 69c655706f674bd790685ede15052973 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69c655706f674bd790685ede15052973 (127.18.211.65). New cstate: current_term: 1 leader_uuid: "69c655706f674bd790685ede15052973" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d86f1e60244c42579e5f90793e1b7ed6" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 41913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04268ae26e9c44b18e0b5c2c095bb2c3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69c655706f674bd790685ede15052973" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 45013 } health_report { overall_health: HEALTHY } } }
W20260928 06:43:51.500547 5193 tablet.cc:2369] T 5024546682c2420db8540c03ee0d3cdc P 04268ae26e9c44b18e0b5c2c095bb2c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:43:51.506685 5343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:51.504640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42758 (local address 127.18.211.65:45013)
0928 06:43:51.504687 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:51.504689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:51.504697 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:43:51.504727 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:51.504728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:51.504743 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:51.504799 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:43:51.504921 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.505373 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.506058 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.506235 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.506304 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:51.506428 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:43:51.506508 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:43:51.506577 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:51.506610 (+ 33us) server_negotiation.cc:299] Negotiation successful
0928 06:43:51.506619 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260928 06:43:51.508327 5281 raft_consensus.cc:1261] T 5024546682c2420db8540c03ee0d3cdc P d86f1e60244c42579e5f90793e1b7ed6 [term 1 FOLLOWER]: Refusing update from remote peer 69c655706f674bd790685ede15052973: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:43:51.508325 5146 raft_consensus.cc:1261] T 5024546682c2420db8540c03ee0d3cdc P 04268ae26e9c44b18e0b5c2c095bb2c3 [term 1 FOLLOWER]: Refusing update from remote peer 69c655706f674bd790685ede15052973: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:43:51.508667 5354 consensus_queue.cc:1035] T 5024546682c2420db8540c03ee0d3cdc P 69c655706f674bd790685ede15052973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04268ae26e9c44b18e0b5c2c095bb2c3" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 33005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:43:51.508783 5351 consensus_queue.cc:1035] T 5024546682c2420db8540c03ee0d3cdc P 69c655706f674bd790685ede15052973 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d86f1e60244c42579e5f90793e1b7ed6" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 41913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:43:51.510898 5357 mvcc.cc:205] Tried to move back new op lower bound from 7334206797855498240 to 7334206797794803712. Current Snapshot: MvccSnapshot[applied={T|T < 7334206797855498240}]
I20260928 06:43:51.511435 5359 mvcc.cc:205] Tried to move back new op lower bound from 7334206797855498240 to 7334206797794803712. Current Snapshot: MvccSnapshot[applied={T|T < 7334206797855498240}]
I20260928 06:43:51.511585 5358 mvcc.cc:205] Tried to move back new op lower bound from 7334206797855498240 to 7334206797794803712. Current Snapshot: MvccSnapshot[applied={T|T < 7334206797855498240}]
I20260928 06:43:51.513513 4863 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:43:51.513621 4863 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:43:51.514272 4863 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d814e3c2ed047a3b241f27f6b45ead1: Sending DeleteTablet for 3 replicas of tablet 5024546682c2420db8540c03ee0d3cdc
I20260928 06:43:51.514585 4990 tablet_service.cc:1514] Processing DeleteTablet for tablet 5024546682c2420db8540c03ee0d3cdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:43:51 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:42746
I20260928 06:43:51.514595 5126 tablet_service.cc:1514] Processing DeleteTablet for tablet 5024546682c2420db8540c03ee0d3cdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:43:51 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:59838
I20260928 06:43:51.514642 5261 tablet_service.cc:1514] Processing DeleteTablet for tablet 5024546682c2420db8540c03ee0d3cdc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:43:51 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:42224
I20260928 06:43:51.514815 5369 tablet_replica.cc:330] T 5024546682c2420db8540c03ee0d3cdc P d86f1e60244c42579e5f90793e1b7ed6: stopping tablet replica
I20260928 06:43:51.514902 5369 raft_consensus.cc:2229] T 5024546682c2420db8540c03ee0d3cdc P d86f1e60244c42579e5f90793e1b7ed6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:43:51.515004 5369 raft_consensus.cc:2258] T 5024546682c2420db8540c03ee0d3cdc P d86f1e60244c42579e5f90793e1b7ed6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:43:51.515025 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 4925
I20260928 06:43:51.515594 5368 tablet_replica.cc:330] T 5024546682c2420db8540c03ee0d3cdc P 04268ae26e9c44b18e0b5c2c095bb2c3: stopping tablet replica
I20260928 06:43:51.515702 5368 raft_consensus.cc:2229] T 5024546682c2420db8540c03ee0d3cdc P 04268ae26e9c44b18e0b5c2c095bb2c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:43:51.515803 5368 raft_consensus.cc:2258] T 5024546682c2420db8540c03ee0d3cdc P 04268ae26e9c44b18e0b5c2c095bb2c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:43:51.516098 5369 ts_tablet_manager.cc:1905] T 5024546682c2420db8540c03ee0d3cdc P d86f1e60244c42579e5f90793e1b7ed6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:43:51.516213 5368 ts_tablet_manager.cc:1905] T 5024546682c2420db8540c03ee0d3cdc P 04268ae26e9c44b18e0b5c2c095bb2c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:43:51.517627 5369 ts_tablet_manager.cc:1918] T 5024546682c2420db8540c03ee0d3cdc P d86f1e60244c42579e5f90793e1b7ed6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:43:51.517688 5368 ts_tablet_manager.cc:1918] T 5024546682c2420db8540c03ee0d3cdc P 04268ae26e9c44b18e0b5c2c095bb2c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:43:51.517704 5369 log.cc:1196] T 5024546682c2420db8540c03ee0d3cdc P d86f1e60244c42579e5f90793e1b7ed6: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-2/wal/wals/5024546682c2420db8540c03ee0d3cdc
I20260928 06:43:51.517755 5368 log.cc:1196] T 5024546682c2420db8540c03ee0d3cdc P 04268ae26e9c44b18e0b5c2c095bb2c3: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790577602522428-19277-0/minicluster-data/ts-1/wal/wals/5024546682c2420db8540c03ee0d3cdc
I20260928 06:43:51.517975 5369 ts_tablet_manager.cc:1939] T 5024546682c2420db8540c03ee0d3cdc P d86f1e60244c42579e5f90793e1b7ed6: Deleting consensus metadata
I20260928 06:43:51.518005 5368 ts_tablet_manager.cc:1939] T 5024546682c2420db8540c03ee0d3cdc P 04268ae26e9c44b18e0b5c2c095bb2c3: Deleting consensus metadata
I20260928 06:43:51.518364 4850 catalog_manager.cc:4841] TS 04268ae26e9c44b18e0b5c2c095bb2c3 (127.18.211.66:33005): tablet 5024546682c2420db8540c03ee0d3cdc (table test-table [id=a93252b5cf474cb1ac2de62680655881]) successfully deleted
I20260928 06:43:51.518435 4849 catalog_manager.cc:4841] TS d86f1e60244c42579e5f90793e1b7ed6 (127.18.211.67:41913): tablet 5024546682c2420db8540c03ee0d3cdc (table test-table [id=a93252b5cf474cb1ac2de62680655881]) successfully deleted
W20260928 06:43:51.521706 4849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.211.65:45013 (error 108)
I20260928 06:43:51.521898 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 5062
I20260928 06:43:51.522151 5338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:51.522046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:45013 (local address 127.0.0.1:42762)
0928 06:43:51.522092 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:51.522120 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.65:45013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260928 06:43:51.522246 4849 catalog_manager.cc:4584] TS 69c655706f674bd790685ede15052973 (127.18.211.65:45013): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5024546682c2420db8540c03ee0d3cdc: Network error: Client connection negotiation failed: client connection to 127.18.211.65:45013: connect: Connection refused (error 111)
I20260928 06:43:51.526212 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 5197
I20260928 06:43:51.530395 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 4831
2026-09-28T06:43:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1777 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T06:43:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:43:51Z Disabled control of system clock
I20260928 06:43:51.556349 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:44331
--webserver_interface=127.18.211.126
--webserver_port=0
--builtin_ntp_servers=127.18.211.84:37769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:44331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:43:51.634763 5374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:43:51.634984 5374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:43:51.635004 5374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:43:51.636150 5374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:43:51.636193 5374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:43:51.636204 5374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:43:51.636215 5374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:43:51.636226 5374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:43:51.637521 5374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:37769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:44331
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:44331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:43:51.637705 5374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:43:51.637950 5374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:43:51.642482 5379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:51.642581 5374 server_base.cc:1034] running on GCE node
W20260928 06:43:51.642491 5380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:51.642513 5382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:51.642930 5374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:43:51.643240 5374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:43:51.644397 5374 hybrid_clock.cc:648] HybridClock initialized: now 1790577831644381 us; error 35 us; skew 500 ppm
I20260928 06:43:51.645551 5374 webserver.cc:460] Webserver started at http://127.18.211.126:38039/ using document root <none> and password file <none>
I20260928 06:43:51.645774 5374 fs_manager.cc:359] Metadata directory not provided
I20260928 06:43:51.645826 5374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:43:51.645951 5374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:43:51.646960 5374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "e6452cc8afd2439e90fa98188f12a0ac"
format_stamp: "Formatted at 2026-09-28 06:43:51 on dist-test-slave-b28r"
server_key: "3f08303c391a036ffd97e7277ee99d3a"
server_key_iv: "3cd99781df6691a4b571b69272c55359"
server_key_version: "encryptionkey@0"
I20260928 06:43:51.647346 5374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "e6452cc8afd2439e90fa98188f12a0ac"
format_stamp: "Formatted at 2026-09-28 06:43:51 on dist-test-slave-b28r"
server_key: "3f08303c391a036ffd97e7277ee99d3a"
server_key_iv: "3cd99781df6691a4b571b69272c55359"
server_key_version: "encryptionkey@0"
I20260928 06:43:51.648532 5374 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:43:51.649140 5388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:51.649317 5374 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:43:51.649396 5374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "e6452cc8afd2439e90fa98188f12a0ac"
format_stamp: "Formatted at 2026-09-28 06:43:51 on dist-test-slave-b28r"
server_key: "3f08303c391a036ffd97e7277ee99d3a"
server_key_iv: "3cd99781df6691a4b571b69272c55359"
server_key_version: "encryptionkey@0"
I20260928 06:43:51.649454 5374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:43:51.658636 5374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:43:51.658929 5374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:43:51.659057 5374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:43:51.662810 5374 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:44331
I20260928 06:43:51.662844 5441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:44331 every 8 connection(s)
I20260928 06:43:51.663203 5374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:43:51.664096 5442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:43:51.666400 5442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6452cc8afd2439e90fa98188f12a0ac: Bootstrap starting.
I20260928 06:43:51.666951 5442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6452cc8afd2439e90fa98188f12a0ac: Neither blocks nor log segments found. Creating new log.
I20260928 06:43:51.667248 5442 log.cc:824] T 00000000000000000000000000000000 P e6452cc8afd2439e90fa98188f12a0ac: Log is configured to *not* fsync() on all Append() calls
I20260928 06:43:51.667896 5442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6452cc8afd2439e90fa98188f12a0ac: No bootstrap required, opened a new log
I20260928 06:43:51.669513 5442 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6452cc8afd2439e90fa98188f12a0ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6452cc8afd2439e90fa98188f12a0ac" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44331 } }
I20260928 06:43:51.669651 5442 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6452cc8afd2439e90fa98188f12a0ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:43:51.669672 5442 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6452cc8afd2439e90fa98188f12a0ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6452cc8afd2439e90fa98188f12a0ac, State: Initialized, Role: FOLLOWER
I20260928 06:43:51.669816 5442 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6452cc8afd2439e90fa98188f12a0ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6452cc8afd2439e90fa98188f12a0ac" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44331 } }
I20260928 06:43:51.669878 5442 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6452cc8afd2439e90fa98188f12a0ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:43:51.669900 5442 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6452cc8afd2439e90fa98188f12a0ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:43:51.669946 5442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6452cc8afd2439e90fa98188f12a0ac [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:43:51.670341 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 5374
I20260928 06:43:51.670457 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:43:51.670571 5442 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6452cc8afd2439e90fa98188f12a0ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6452cc8afd2439e90fa98188f12a0ac" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44331 } }
I20260928 06:43:51.670668 19277 external_mini_cluster.cc:1428] Setting key 15221a1613302945d7bdcd0d54c3b710
I20260928 06:43:51.670677 5442 leader_election.cc:302] T 00000000000000000000000000000000 P e6452cc8afd2439e90fa98188f12a0ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6452cc8afd2439e90fa98188f12a0ac; no voters:
I20260928 06:43:51.670849 5442 leader_election.cc:288] T 00000000000000000000000000000000 P e6452cc8afd2439e90fa98188f12a0ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:43:51.670899 5445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6452cc8afd2439e90fa98188f12a0ac [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:43:51.671052 5445 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6452cc8afd2439e90fa98188f12a0ac [term 1 LEADER]: Becoming Leader. State: Replica: e6452cc8afd2439e90fa98188f12a0ac, State: Running, Role: LEADER
I20260928 06:43:51.671132 5442 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6452cc8afd2439e90fa98188f12a0ac [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:43:51.671182 5445 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6452cc8afd2439e90fa98188f12a0ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6452cc8afd2439e90fa98188f12a0ac" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44331 } }
I20260928 06:43:51.671525 5447 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6452cc8afd2439e90fa98188f12a0ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6452cc8afd2439e90fa98188f12a0ac. Latest consensus state: current_term: 1 leader_uuid: "e6452cc8afd2439e90fa98188f12a0ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6452cc8afd2439e90fa98188f12a0ac" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44331 } } }
I20260928 06:43:51.671605 5447 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6452cc8afd2439e90fa98188f12a0ac [sys.catalog]: This master's current role is: LEADER
I20260928 06:43:51.671692 5446 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6452cc8afd2439e90fa98188f12a0ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6452cc8afd2439e90fa98188f12a0ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6452cc8afd2439e90fa98188f12a0ac" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 44331 } } }
I20260928 06:43:51.671798 5454 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:43:51.671854 5446 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6452cc8afd2439e90fa98188f12a0ac [sys.catalog]: This master's current role is: LEADER
I20260928 06:43:51.672818 5454 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:43:51.674933 5454 catalog_manager.cc:1347] Generated new cluster ID: 283c2703782e4bad869a7b2c5073fd85
I20260928 06:43:51.674991 5454 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:43:51.676968 5450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:51.671399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58092 (local address 127.18.211.126:44331)
0928 06:43:51.671604 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:51.671613 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:51.672124 (+ 511us) server_negotiation.cc:407] Connection header received
0928 06:43:51.672276 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:51.672295 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:51.672368 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:51.672510 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:51.672693 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.673927 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.674880 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.675479 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.675592 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:51.676283 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:51.676288 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:51.676288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:43:51.676483 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:51.676524 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:43:51.676538 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:51.676598 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:51.676675 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 06:43:51.676751 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":64,"threads_started":1}
I20260928 06:43:51.697634 5454 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:43:51.698387 5454 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:43:51.705288 5454 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6452cc8afd2439e90fa98188f12a0ac: Generated new TSK 0
I20260928 06:43:51.705488 5454 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:43:51.709147 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.65:0
--local_ip_for_outbound_sockets=127.18.211.65
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:44331
--builtin_ntp_servers=127.18.211.84:37769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:43:51.788082 5466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:43:51.788288 5466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:43:51.788319 5466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:43:51.789441 5466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:43:51.789480 5466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:43:51.789513 5466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.65
I20260928 06:43:51.790850 5466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:37769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:44331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.65
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:43:51.791097 5466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:43:51.791407 5466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:43:51.796923 5471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:51.796938 5472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:51.797008 5474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:51.797245 5466 server_base.cc:1034] running on GCE node
I20260928 06:43:51.797475 5466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:43:51.797701 5466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:43:51.798852 5466 hybrid_clock.cc:648] HybridClock initialized: now 1790577831798835 us; error 27 us; skew 500 ppm
I20260928 06:43:51.800140 5466 webserver.cc:460] Webserver started at http://127.18.211.65:43225/ using document root <none> and password file <none>
I20260928 06:43:51.800380 5466 fs_manager.cc:359] Metadata directory not provided
I20260928 06:43:51.800431 5466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:43:51.800555 5466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:43:51.801587 5466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-0/data/instance:
uuid: "5ec43ba041ec4f5fa04788eea02b90ad"
format_stamp: "Formatted at 2026-09-28 06:43:51 on dist-test-slave-b28r"
server_key: "0afc00f3291fa8c5830f99eaf653e67e"
server_key_iv: "09172a924c3eb17b7cf3989554c93c94"
server_key_version: "encryptionkey@0"
I20260928 06:43:51.801908 5466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance:
uuid: "5ec43ba041ec4f5fa04788eea02b90ad"
format_stamp: "Formatted at 2026-09-28 06:43:51 on dist-test-slave-b28r"
server_key: "0afc00f3291fa8c5830f99eaf653e67e"
server_key_iv: "09172a924c3eb17b7cf3989554c93c94"
server_key_version: "encryptionkey@0"
I20260928 06:43:51.803171 5466 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:43:51.803913 5480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:51.804057 5466 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:43:51.804136 5466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-0/wal
uuid: "5ec43ba041ec4f5fa04788eea02b90ad"
format_stamp: "Formatted at 2026-09-28 06:43:51 on dist-test-slave-b28r"
server_key: "0afc00f3291fa8c5830f99eaf653e67e"
server_key_iv: "09172a924c3eb17b7cf3989554c93c94"
server_key_version: "encryptionkey@0"
I20260928 06:43:51.804203 5466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:43:51.833680 5466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:43:51.834012 5466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:43:51.834147 5466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:43:51.834379 5466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:43:51.834702 5466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:43:51.834735 5466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:51.834764 5466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:43:51.834781 5466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:51.840313 5466 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.65:37919
I20260928 06:43:51.840401 5594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.65:37919 every 8 connection(s)
I20260928 06:43:51.840685 5466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
I20260928 06:43:51.843988 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 5466
I20260928 06:43:51.844091 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance
I20260928 06:43:51.844295 19277 external_mini_cluster.cc:1428] Setting key 20d62ad9033582efa925b3c0dc79cc54
I20260928 06:43:51.845372 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.66:0
--local_ip_for_outbound_sockets=127.18.211.66
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:44331
--builtin_ntp_servers=127.18.211.84:37769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:43:51.846108 5450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:51.841368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:58431 (local address 127.18.211.126:44331)
0928 06:43:51.841418 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:51.841421 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:51.841649 (+ 228us) server_negotiation.cc:407] Connection header received
0928 06:43:51.842129 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:51.842130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:51.842141 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:51.842188 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:51.842863 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.843411 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.844635 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.844913 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.844977 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:51.845879 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:51.845880 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:51.845881 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:43:51.845885 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:51.845900 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:43:51.845903 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:51.845943 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:51.846025 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 06:43:51.846051 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:43:51.846346 5597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:51.841358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44331 (local address 127.18.211.65:58431)
0928 06:43:51.841534 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:51.841568 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:51.842052 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:51.842250 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:51.842253 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:51.842263 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:43:51.842797 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:51.842803 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.843478 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.843479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:51.844538 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:51.844543 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.844990 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.844991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:51.845139 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:51.845706 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:43:51.845707 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:43:51.845748 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:43:51.845806 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:43:51.845957 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:43:51.845961 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:43:51.845964 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:43:51.846027 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 06:43:51.846131 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":84,"threads_started":1}
I20260928 06:43:51.846866 5595 heartbeater.cc:347] Connected to a master server at 127.18.211.126:44331
I20260928 06:43:51.846958 5595 heartbeater.cc:464] Registering TS with master...
I20260928 06:43:51.847175 5595 heartbeater.cc:511] Master 127.18.211.126:44331 requested a full tablet report, sending...
I20260928 06:43:51.847662 5406 ts_manager.cc:194] Registered new tserver with Master: 5ec43ba041ec4f5fa04788eea02b90ad (127.18.211.65:37919)
I20260928 06:43:51.848272 5406 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.211.65:58431
W20260928 06:43:51.928270 5598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:43:51.928483 5598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:43:51.928503 5598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:43:51.929592 5598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:43:51.929622 5598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:43:51.929646 5598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.66
I20260928 06:43:51.930907 5598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:37769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:44331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.66
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:43:51.931123 5598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:43:51.931401 5598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:43:51.936184 5604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:51.936235 5606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:51.936252 5603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:51.936511 5598 server_base.cc:1034] running on GCE node
I20260928 06:43:51.936688 5598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:43:51.936904 5598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:43:51.938071 5598 hybrid_clock.cc:648] HybridClock initialized: now 1790577831938053 us; error 30 us; skew 500 ppm
I20260928 06:43:51.943578 5598 webserver.cc:460] Webserver started at http://127.18.211.66:33267/ using document root <none> and password file <none>
I20260928 06:43:51.943861 5598 fs_manager.cc:359] Metadata directory not provided
I20260928 06:43:51.943913 5598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:43:51.944036 5598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:43:51.945127 5598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-1/data/instance:
uuid: "dfe3c2216efc4458baec32c0bfe565a9"
format_stamp: "Formatted at 2026-09-28 06:43:51 on dist-test-slave-b28r"
server_key: "032a2f921b865e33f3c7de2052c93366"
server_key_iv: "7f696d0853763b25ac7ce0e18d113238"
server_key_version: "encryptionkey@0"
I20260928 06:43:51.945451 5598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance:
uuid: "dfe3c2216efc4458baec32c0bfe565a9"
format_stamp: "Formatted at 2026-09-28 06:43:51 on dist-test-slave-b28r"
server_key: "032a2f921b865e33f3c7de2052c93366"
server_key_iv: "7f696d0853763b25ac7ce0e18d113238"
server_key_version: "encryptionkey@0"
I20260928 06:43:51.946676 5598 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:43:51.947516 5612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:51.947692 5598 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:43:51.947778 5598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-1/wal
uuid: "dfe3c2216efc4458baec32c0bfe565a9"
format_stamp: "Formatted at 2026-09-28 06:43:51 on dist-test-slave-b28r"
server_key: "032a2f921b865e33f3c7de2052c93366"
server_key_iv: "7f696d0853763b25ac7ce0e18d113238"
server_key_version: "encryptionkey@0"
I20260928 06:43:51.947839 5598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:43:51.964563 5598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:43:51.964833 5598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:43:51.964956 5598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:43:51.965162 5598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:43:51.965456 5598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:43:51.965489 5598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:51.965516 5598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:43:51.965534 5598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:51.970829 5598 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.66:39587
I20260928 06:43:51.970927 5725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.66:39587 every 8 connection(s)
I20260928 06:43:51.971238 5598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
I20260928 06:43:51.971395 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 5598
I20260928 06:43:51.971503 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance
I20260928 06:43:51.971726 19277 external_mini_cluster.cc:1428] Setting key 290005b831ac7419d9edf40a78e3194c
I20260928 06:43:51.972875 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.67:0
--local_ip_for_outbound_sockets=127.18.211.67
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:44331
--builtin_ntp_servers=127.18.211.84:37769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:43:51.977118 5450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:51.971980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:42477 (local address 127.18.211.126:44331)
0928 06:43:51.972042 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:51.972044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:51.972621 (+ 577us) server_negotiation.cc:407] Connection header received
0928 06:43:51.973121 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:51.973123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:51.973136 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:51.973207 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:51.973827 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.974443 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.975633 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.975919 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.976008 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:51.976759 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:51.976760 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:51.976761 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:43:51.976767 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:51.976787 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:43:51.976792 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:51.976866 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:51.976980 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 06:43:51.977016 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:43:51.977286 5728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:51.972022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44331 (local address 127.18.211.66:42477)
0928 06:43:51.972443 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:51.972487 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:51.973020 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:51.973257 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:51.973260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:51.973268 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:43:51.973741 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:51.973745 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.974554 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.974555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:51.975542 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:51.975545 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:51.976008 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:51.976009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:51.976115 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:51.976613 (+ 498us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:43:51.976614 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:43:51.976641 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:43:51.976664 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:43:51.976890 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:43:51.976897 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:43:51.976900 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:43:51.976971 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 06:43:51.977053 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":97,"threads_started":1}
I20260928 06:43:51.977771 5726 heartbeater.cc:347] Connected to a master server at 127.18.211.126:44331
I20260928 06:43:51.977876 5726 heartbeater.cc:464] Registering TS with master...
I20260928 06:43:51.978077 5726 heartbeater.cc:511] Master 127.18.211.126:44331 requested a full tablet report, sending...
I20260928 06:43:51.978479 5406 ts_manager.cc:194] Registered new tserver with Master: dfe3c2216efc4458baec32c0bfe565a9 (127.18.211.66:39587)
I20260928 06:43:51.978842 5406 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.211.66:42477
W20260928 06:43:52.054121 5729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:43:52.054324 5729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:43:52.054344 5729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:43:52.055483 5729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:43:52.055524 5729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:43:52.055548 5729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.67
I20260928 06:43:52.056874 5729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:37769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:44331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.67
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:43:52.057082 5729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:43:52.057380 5729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:43:52.062007 5734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:52.062023 5737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:52.062024 5735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:52.062227 5729 server_base.cc:1034] running on GCE node
I20260928 06:43:52.062475 5729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:43:52.062686 5729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:43:52.063827 5729 hybrid_clock.cc:648] HybridClock initialized: now 1790577832063811 us; error 30 us; skew 500 ppm
I20260928 06:43:52.065049 5729 webserver.cc:460] Webserver started at http://127.18.211.67:42343/ using document root <none> and password file <none>
I20260928 06:43:52.065253 5729 fs_manager.cc:359] Metadata directory not provided
I20260928 06:43:52.065330 5729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:43:52.065456 5729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:43:52.066466 5729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-2/data/instance:
uuid: "2b61025b43494dc8996fe1d6d32ebb15"
format_stamp: "Formatted at 2026-09-28 06:43:52 on dist-test-slave-b28r"
server_key: "2450c2e3bd5cc30d5be518e5fa19f2f0"
server_key_iv: "7d566f73b8eae23689bee28f758c4aff"
server_key_version: "encryptionkey@0"
I20260928 06:43:52.066788 5729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance:
uuid: "2b61025b43494dc8996fe1d6d32ebb15"
format_stamp: "Formatted at 2026-09-28 06:43:52 on dist-test-slave-b28r"
server_key: "2450c2e3bd5cc30d5be518e5fa19f2f0"
server_key_iv: "7d566f73b8eae23689bee28f758c4aff"
server_key_version: "encryptionkey@0"
I20260928 06:43:52.068091 5729 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:43:52.068805 5743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:52.068948 5729 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:43:52.069027 5729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-2/wal
uuid: "2b61025b43494dc8996fe1d6d32ebb15"
format_stamp: "Formatted at 2026-09-28 06:43:52 on dist-test-slave-b28r"
server_key: "2450c2e3bd5cc30d5be518e5fa19f2f0"
server_key_iv: "7d566f73b8eae23689bee28f758c4aff"
server_key_version: "encryptionkey@0"
I20260928 06:43:52.069083 5729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:43:52.078828 5729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:43:52.079104 5729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:43:52.079233 5729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:43:52.079428 5729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:43:52.079744 5729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:43:52.079785 5729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:52.079813 5729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:43:52.079836 5729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:52.085150 5729 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.67:40491
I20260928 06:43:52.085204 5857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.67:40491 every 8 connection(s)
I20260928 06:43:52.085495 5729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
I20260928 06:43:52.087721 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 5729
I20260928 06:43:52.087819 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance
I20260928 06:43:52.088013 19277 external_mini_cluster.cc:1428] Setting key 0e7ae8c99776e92771cf32cfd033d8da
I20260928 06:43:52.090438 5450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:52.086121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:57015 (local address 127.18.211.126:44331)
0928 06:43:52.086180 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:52.086182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:52.086409 (+ 227us) server_negotiation.cc:407] Connection header received
0928 06:43:52.086874 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:52.086875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:52.086886 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:52.086944 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:52.087624 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:52.088224 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:52.089178 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:52.089398 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:52.089454 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:52.090185 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:52.090186 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:52.090186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:43:52.090190 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:52.090205 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:43:52.090209 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:52.090253 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:52.090330 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 06:43:52.090360 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 06:43:52.090629 5860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:52.086101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:44331 (local address 127.18.211.67:57015)
0928 06:43:52.086284 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:52.086324 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:52.086791 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:52.087004 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:52.087008 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:52.087015 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:43:52.087522 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:52.087527 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:52.088313 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:52.088314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:52.089101 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:52.089103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:52.089472 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:52.089473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:52.089597 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:52.090063 (+ 466us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:43:52.090064 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:43:52.090090 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:43:52.090111 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:43:52.090260 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:43:52.090266 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:43:52.090269 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:43:52.090330 (+ 61us) client_negotiation.cc:240] Negotiation successful
0928 06:43:52.090426 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
I20260928 06:43:52.091333 5858 heartbeater.cc:347] Connected to a master server at 127.18.211.126:44331
I20260928 06:43:52.091415 5858 heartbeater.cc:464] Registering TS with master...
I20260928 06:43:52.091588 5858 heartbeater.cc:511] Master 127.18.211.126:44331 requested a full tablet report, sending...
I20260928 06:43:52.091959 5406 ts_manager.cc:194] Registered new tserver with Master: 2b61025b43494dc8996fe1d6d32ebb15 (127.18.211.67:40491)
I20260928 06:43:52.092346 5406 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.211.67:57015
I20260928 06:43:52.099682 19277 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:43:52.103991 5450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:52.100974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58096 (local address 127.18.211.126:44331)
0928 06:43:52.101042 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:52.101044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:52.101176 (+ 132us) server_negotiation.cc:407] Connection header received
0928 06:43:52.101200 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:52.101200 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:52.101211 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:52.101268 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:52.101438 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:52.101912 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:52.102701 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:52.102902 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:52.102956 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:52.103738 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:52.103739 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:52.103739 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:43:52.103743 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:52.103757 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:43:52.103760 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:52.103841 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:52.103909 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 06:43:52.103919 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 06:43:52.105481 5406 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58096:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:43:52.107030 5406 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:43:52.113296 5867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:52.109262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:37919 (local address 127.0.0.1:36022)
0928 06:43:52.109562 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:52.109594 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:52.109665 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:52.109899 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:52.109902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:52.109904 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:43:52.109972 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:52.109974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:52.111320 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:52.111321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:52.112352 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:52.112354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:52.113007 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:52.113009 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:52.113083 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:52.113097 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 06:43:52.113173 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 06:43:52.113222 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":86,"threads_started":1}
I20260928 06:43:52.113523 5871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:52.109520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36022 (local address 127.18.211.65:37919)
0928 06:43:52.109719 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:52.109724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:52.109735 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:43:52.109778 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:52.109781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:52.109797 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:52.109886 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:43:52.110068 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:52.111231 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:52.112429 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:52.112924 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:52.112988 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:52.113082 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:52.113407 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 06:43:52.113451 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":87,"threads_started":1}
I20260928 06:43:52.113552 5866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:52.109287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:40491 (local address 127.0.0.1:53700)
0928 06:43:52.109985 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:52.109995 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:52.110046 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:52.110282 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:52.110282 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:52.110285 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:43:52.110319 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:52.110320 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:52.111505 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:52.111507 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:52.112361 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:52.112364 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:52.113324 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:52.113325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:52.113386 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:52.113391 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:43:52.113445 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 06:43:52.113484 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":89,"threads_started":1}
I20260928 06:43:52.113552 5869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:52.109344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50776 (local address 127.18.211.66:39587)
0928 06:43:52.109788 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:52.109791 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:52.109799 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:43:52.109831 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:52.109833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:52.109848 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:52.109922 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:43:52.110119 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:52.110909 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:52.112329 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:52.113088 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:52.113182 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:52.113275 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:52.113316 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 06:43:52.113460 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":112,"threads_started":1}
I20260928 06:43:52.113853 5870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:52.109508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53700 (local address 127.18.211.67:40491)
0928 06:43:52.109694 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:52.109711 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:52.110112 (+ 401us) server_negotiation.cc:407] Connection header received
0928 06:43:52.110171 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:52.110174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:52.110189 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:52.110270 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:43:52.110383 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:52.111430 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:52.112461 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:52.113152 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:52.113214 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:52.113285 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:52.113736 (+ 451us) server_negotiation.cc:299] Negotiation successful
0928 06:43:52.113771 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":86,"threads_started":1}
I20260928 06:43:52.113962 5868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:52.109243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:39587 (local address 127.0.0.1:50776)
0928 06:43:52.109534 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:52.109576 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:52.109638 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:52.109996 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:52.109997 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:52.109999 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:43:52.110044 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:52.110046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:52.110984 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:52.110984 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:52.112242 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:52.112245 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:52.113186 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:52.113187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:52.113248 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:52.113250 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:43:52.113286 (+ 36us) client_negotiation.cc:240] Negotiation successful
0928 06:43:52.113315 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":87,"threads_started":1}
I20260928 06:43:52.114281 5528 tablet_service.cc:1467] Processing CreateTablet for tablet f1e77641d7ee46d9a9badafead0c10f2 (DEFAULT_TABLE table=test-table [id=bae64236b6004fbe9d4f3509443496d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:43:52.114439 5791 tablet_service.cc:1467] Processing CreateTablet for tablet f1e77641d7ee46d9a9badafead0c10f2 (DEFAULT_TABLE table=test-table [id=bae64236b6004fbe9d4f3509443496d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:43:52.114686 5528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1e77641d7ee46d9a9badafead0c10f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:43:52.114604 5660 tablet_service.cc:1467] Processing CreateTablet for tablet f1e77641d7ee46d9a9badafead0c10f2 (DEFAULT_TABLE table=test-table [id=bae64236b6004fbe9d4f3509443496d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:43:52.114751 5791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1e77641d7ee46d9a9badafead0c10f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:43:52.114930 5660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1e77641d7ee46d9a9badafead0c10f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:43:52.117488 5872 tablet_bootstrap.cc:492] T f1e77641d7ee46d9a9badafead0c10f2 P dfe3c2216efc4458baec32c0bfe565a9: Bootstrap starting.
I20260928 06:43:52.118163 5872 tablet_bootstrap.cc:654] T f1e77641d7ee46d9a9badafead0c10f2 P dfe3c2216efc4458baec32c0bfe565a9: Neither blocks nor log segments found. Creating new log.
I20260928 06:43:52.118280 5873 tablet_bootstrap.cc:492] T f1e77641d7ee46d9a9badafead0c10f2 P 5ec43ba041ec4f5fa04788eea02b90ad: Bootstrap starting.
I20260928 06:43:52.118340 5874 tablet_bootstrap.cc:492] T f1e77641d7ee46d9a9badafead0c10f2 P 2b61025b43494dc8996fe1d6d32ebb15: Bootstrap starting.
I20260928 06:43:52.118548 5872 log.cc:824] T f1e77641d7ee46d9a9badafead0c10f2 P dfe3c2216efc4458baec32c0bfe565a9: Log is configured to *not* fsync() on all Append() calls
I20260928 06:43:52.118966 5874 tablet_bootstrap.cc:654] T f1e77641d7ee46d9a9badafead0c10f2 P 2b61025b43494dc8996fe1d6d32ebb15: Neither blocks nor log segments found. Creating new log.
I20260928 06:43:52.118966 5873 tablet_bootstrap.cc:654] T f1e77641d7ee46d9a9badafead0c10f2 P 5ec43ba041ec4f5fa04788eea02b90ad: Neither blocks nor log segments found. Creating new log.
I20260928 06:43:52.119263 5872 tablet_bootstrap.cc:492] T f1e77641d7ee46d9a9badafead0c10f2 P dfe3c2216efc4458baec32c0bfe565a9: No bootstrap required, opened a new log
I20260928 06:43:52.119323 5872 ts_tablet_manager.cc:1397] T f1e77641d7ee46d9a9badafead0c10f2 P dfe3c2216efc4458baec32c0bfe565a9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:43:52.119329 5874 log.cc:824] T f1e77641d7ee46d9a9badafead0c10f2 P 2b61025b43494dc8996fe1d6d32ebb15: Log is configured to *not* fsync() on all Append() calls
I20260928 06:43:52.119329 5873 log.cc:824] T f1e77641d7ee46d9a9badafead0c10f2 P 5ec43ba041ec4f5fa04788eea02b90ad: Log is configured to *not* fsync() on all Append() calls
I20260928 06:43:52.120223 5874 tablet_bootstrap.cc:492] T f1e77641d7ee46d9a9badafead0c10f2 P 2b61025b43494dc8996fe1d6d32ebb15: No bootstrap required, opened a new log
I20260928 06:43:52.120302 5874 ts_tablet_manager.cc:1397] T f1e77641d7ee46d9a9badafead0c10f2 P 2b61025b43494dc8996fe1d6d32ebb15: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 06:43:52.120324 5873 tablet_bootstrap.cc:492] T f1e77641d7ee46d9a9badafead0c10f2 P 5ec43ba041ec4f5fa04788eea02b90ad: No bootstrap required, opened a new log
I20260928 06:43:52.120378 5873 ts_tablet_manager.cc:1397] T f1e77641d7ee46d9a9badafead0c10f2 P 5ec43ba041ec4f5fa04788eea02b90ad: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 06:43:52.121367 5872 raft_consensus.cc:348] T f1e77641d7ee46d9a9badafead0c10f2 P dfe3c2216efc4458baec32c0bfe565a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfe3c2216efc4458baec32c0bfe565a9" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 39587 } } peers { permanent_uuid: "5ec43ba041ec4f5fa04788eea02b90ad" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 37919 } } peers { permanent_uuid: "2b61025b43494dc8996fe1d6d32ebb15" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40491 } }
I20260928 06:43:52.121548 5872 raft_consensus.cc:374] T f1e77641d7ee46d9a9badafead0c10f2 P dfe3c2216efc4458baec32c0bfe565a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:43:52.121590 5872 raft_consensus.cc:729] T f1e77641d7ee46d9a9badafead0c10f2 P dfe3c2216efc4458baec32c0bfe565a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfe3c2216efc4458baec32c0bfe565a9, State: Initialized, Role: FOLLOWER
I20260928 06:43:52.121721 5872 consensus_queue.cc:260] T f1e77641d7ee46d9a9badafead0c10f2 P dfe3c2216efc4458baec32c0bfe565a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfe3c2216efc4458baec32c0bfe565a9" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 39587 } } peers { permanent_uuid: "5ec43ba041ec4f5fa04788eea02b90ad" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 37919 } } peers { permanent_uuid: "2b61025b43494dc8996fe1d6d32ebb15" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40491 } }
I20260928 06:43:52.122005 5726 heartbeater.cc:503] Master 127.18.211.126:44331 was elected leader, sending a full tablet report...
I20260928 06:43:52.122229 5872 ts_tablet_manager.cc:1428] T f1e77641d7ee46d9a9badafead0c10f2 P dfe3c2216efc4458baec32c0bfe565a9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:43:52.122880 5874 raft_consensus.cc:348] T f1e77641d7ee46d9a9badafead0c10f2 P 2b61025b43494dc8996fe1d6d32ebb15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfe3c2216efc4458baec32c0bfe565a9" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 39587 } } peers { permanent_uuid: "5ec43ba041ec4f5fa04788eea02b90ad" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 37919 } } peers { permanent_uuid: "2b61025b43494dc8996fe1d6d32ebb15" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40491 } }
I20260928 06:43:52.122880 5873 raft_consensus.cc:348] T f1e77641d7ee46d9a9badafead0c10f2 P 5ec43ba041ec4f5fa04788eea02b90ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfe3c2216efc4458baec32c0bfe565a9" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 39587 } } peers { permanent_uuid: "5ec43ba041ec4f5fa04788eea02b90ad" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 37919 } } peers { permanent_uuid: "2b61025b43494dc8996fe1d6d32ebb15" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40491 } }
I20260928 06:43:52.123042 5874 raft_consensus.cc:374] T f1e77641d7ee46d9a9badafead0c10f2 P 2b61025b43494dc8996fe1d6d32ebb15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:43:52.123042 5873 raft_consensus.cc:374] T f1e77641d7ee46d9a9badafead0c10f2 P 5ec43ba041ec4f5fa04788eea02b90ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:43:52.123111 5873 raft_consensus.cc:729] T f1e77641d7ee46d9a9badafead0c10f2 P 5ec43ba041ec4f5fa04788eea02b90ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ec43ba041ec4f5fa04788eea02b90ad, State: Initialized, Role: FOLLOWER
I20260928 06:43:52.123123 5874 raft_consensus.cc:729] T f1e77641d7ee46d9a9badafead0c10f2 P 2b61025b43494dc8996fe1d6d32ebb15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b61025b43494dc8996fe1d6d32ebb15, State: Initialized, Role: FOLLOWER
I20260928 06:43:52.123239 5873 consensus_queue.cc:260] T f1e77641d7ee46d9a9badafead0c10f2 P 5ec43ba041ec4f5fa04788eea02b90ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfe3c2216efc4458baec32c0bfe565a9" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 39587 } } peers { permanent_uuid: "5ec43ba041ec4f5fa04788eea02b90ad" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 37919 } } peers { permanent_uuid: "2b61025b43494dc8996fe1d6d32ebb15" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40491 } }
I20260928 06:43:52.123267 5874 consensus_queue.cc:260] T f1e77641d7ee46d9a9badafead0c10f2 P 2b61025b43494dc8996fe1d6d32ebb15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfe3c2216efc4458baec32c0bfe565a9" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 39587 } } peers { permanent_uuid: "5ec43ba041ec4f5fa04788eea02b90ad" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 37919 } } peers { permanent_uuid: "2b61025b43494dc8996fe1d6d32ebb15" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40491 } }
I20260928 06:43:52.123524 5873 ts_tablet_manager.cc:1428] T f1e77641d7ee46d9a9badafead0c10f2 P 5ec43ba041ec4f5fa04788eea02b90ad: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260928 06:43:52.123524 5874 ts_tablet_manager.cc:1428] T f1e77641d7ee46d9a9badafead0c10f2 P 2b61025b43494dc8996fe1d6d32ebb15: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260928 06:43:52.123559 5595 heartbeater.cc:503] Master 127.18.211.126:44331 was elected leader, sending a full tablet report...
I20260928 06:43:52.123633 5858 heartbeater.cc:503] Master 127.18.211.126:44331 was elected leader, sending a full tablet report...
I20260928 06:43:52.156525 5878 raft_consensus.cc:482] T f1e77641d7ee46d9a9badafead0c10f2 P dfe3c2216efc4458baec32c0bfe565a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:43:52.156651 5878 raft_consensus.cc:504] T f1e77641d7ee46d9a9badafead0c10f2 P dfe3c2216efc4458baec32c0bfe565a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfe3c2216efc4458baec32c0bfe565a9" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 39587 } } peers { permanent_uuid: "5ec43ba041ec4f5fa04788eea02b90ad" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 37919 } } peers { permanent_uuid: "2b61025b43494dc8996fe1d6d32ebb15" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40491 } }
I20260928 06:43:52.157042 5878 leader_election.cc:288] T f1e77641d7ee46d9a9badafead0c10f2 P dfe3c2216efc4458baec32c0bfe565a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ec43ba041ec4f5fa04788eea02b90ad (127.18.211.65:37919), 2b61025b43494dc8996fe1d6d32ebb15 (127.18.211.67:40491)
I20260928 06:43:52.161055 5728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:52.157235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:37919 (local address 127.18.211.66:36729)
0928 06:43:52.157281 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:52.157293 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:52.157335 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:52.157539 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:52.157541 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:52.157543 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:43:52.157624 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:52.157627 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:52.158609 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:52.158611 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:52.159977 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:52.159980 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:52.160759 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:52.160761 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:52.160849 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:52.160853 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:43:52.160913 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 06:43:52.160963 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":27}
I20260928 06:43:52.161075 5871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:52.157336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:36729 (local address 127.18.211.65:37919)
0928 06:43:52.157408 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:52.157410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:52.157420 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:43:52.157448 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:52.157449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:52.157461 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:52.157516 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:43:52.157704 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:52.158510 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:52.160092 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:52.160679 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:52.160746 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:52.160754 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:52.160952 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 06:43:52.161003 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 06:43:52.161075 5870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:52.157333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:47053 (local address 127.18.211.67:40491)
0928 06:43:52.157383 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:52.157385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:52.157670 (+ 285us) server_negotiation.cc:407] Connection header received
0928 06:43:52.157742 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:52.157743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:52.157752 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:52.157798 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:43:52.157911 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:52.158572 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:52.160081 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:52.160656 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:52.160745 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:52.160754 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:52.160952 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 06:43:52.161003 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260928 06:43:52.161060 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:52.157235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:40491 (local address 127.18.211.66:47053)
0928 06:43:52.157588 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:52.157598 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:52.157665 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:52.157800 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:52.157800 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:52.157801 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:43:52.157854 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:52.157855 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:52.158657 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:52.158658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:52.159983 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:52.159984 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:52.160760 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:52.160761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:52.160850 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:52.160854 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:43:52.160913 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 06:43:52.160961 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":176,"thread_start_us":135,"threads_started":1}
I20260928 06:43:52.161737 5549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e77641d7ee46d9a9badafead0c10f2" candidate_uuid: "dfe3c2216efc4458baec32c0bfe565a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ec43ba041ec4f5fa04788eea02b90ad" is_pre_election: true
I20260928 06:43:52.161720 5811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e77641d7ee46d9a9badafead0c10f2" candidate_uuid: "dfe3c2216efc4458baec32c0bfe565a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b61025b43494dc8996fe1d6d32ebb15" is_pre_election: true
I20260928 06:43:52.161891 5549 raft_consensus.cc:2454] T f1e77641d7ee46d9a9badafead0c10f2 P 5ec43ba041ec4f5fa04788eea02b90ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfe3c2216efc4458baec32c0bfe565a9 in term 0.
I20260928 06:43:52.161891 5811 raft_consensus.cc:2454] T f1e77641d7ee46d9a9badafead0c10f2 P 2b61025b43494dc8996fe1d6d32ebb15 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dfe3c2216efc4458baec32c0bfe565a9 in term 0.
I20260928 06:43:52.162127 5613 leader_election.cc:302] T f1e77641d7ee46d9a9badafead0c10f2 P dfe3c2216efc4458baec32c0bfe565a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ec43ba041ec4f5fa04788eea02b90ad, dfe3c2216efc4458baec32c0bfe565a9; no voters:
I20260928 06:43:52.162297 5878 raft_consensus.cc:2781] T f1e77641d7ee46d9a9badafead0c10f2 P dfe3c2216efc4458baec32c0bfe565a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:43:52.162339 5878 raft_consensus.cc:482] T f1e77641d7ee46d9a9badafead0c10f2 P dfe3c2216efc4458baec32c0bfe565a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:43:52.162355 5878 raft_consensus.cc:3037] T f1e77641d7ee46d9a9badafead0c10f2 P dfe3c2216efc4458baec32c0bfe565a9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:43:52.163064 5878 raft_consensus.cc:504] T f1e77641d7ee46d9a9badafead0c10f2 P dfe3c2216efc4458baec32c0bfe565a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfe3c2216efc4458baec32c0bfe565a9" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 39587 } } peers { permanent_uuid: "5ec43ba041ec4f5fa04788eea02b90ad" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 37919 } } peers { permanent_uuid: "2b61025b43494dc8996fe1d6d32ebb15" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40491 } }
I20260928 06:43:52.163285 5878 leader_election.cc:288] T f1e77641d7ee46d9a9badafead0c10f2 P dfe3c2216efc4458baec32c0bfe565a9 [CANDIDATE]: Term 1 election: Requested vote from peers 5ec43ba041ec4f5fa04788eea02b90ad (127.18.211.65:37919), 2b61025b43494dc8996fe1d6d32ebb15 (127.18.211.67:40491)
I20260928 06:43:52.163347 5549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e77641d7ee46d9a9badafead0c10f2" candidate_uuid: "dfe3c2216efc4458baec32c0bfe565a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ec43ba041ec4f5fa04788eea02b90ad"
I20260928 06:43:52.163434 5549 raft_consensus.cc:3037] T f1e77641d7ee46d9a9badafead0c10f2 P 5ec43ba041ec4f5fa04788eea02b90ad [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:43:52.163476 5811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1e77641d7ee46d9a9badafead0c10f2" candidate_uuid: "dfe3c2216efc4458baec32c0bfe565a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b61025b43494dc8996fe1d6d32ebb15"
I20260928 06:43:52.163569 5811 raft_consensus.cc:3037] T f1e77641d7ee46d9a9badafead0c10f2 P 2b61025b43494dc8996fe1d6d32ebb15 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:43:52.164170 5549 raft_consensus.cc:2454] T f1e77641d7ee46d9a9badafead0c10f2 P 5ec43ba041ec4f5fa04788eea02b90ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfe3c2216efc4458baec32c0bfe565a9 in term 1.
I20260928 06:43:52.164320 5811 raft_consensus.cc:2454] T f1e77641d7ee46d9a9badafead0c10f2 P 2b61025b43494dc8996fe1d6d32ebb15 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dfe3c2216efc4458baec32c0bfe565a9 in term 1.
I20260928 06:43:52.164325 5613 leader_election.cc:302] T f1e77641d7ee46d9a9badafead0c10f2 P dfe3c2216efc4458baec32c0bfe565a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5ec43ba041ec4f5fa04788eea02b90ad, dfe3c2216efc4458baec32c0bfe565a9; no voters:
I20260928 06:43:52.164495 5878 raft_consensus.cc:2781] T f1e77641d7ee46d9a9badafead0c10f2 P dfe3c2216efc4458baec32c0bfe565a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:43:52.164654 5878 raft_consensus.cc:686] T f1e77641d7ee46d9a9badafead0c10f2 P dfe3c2216efc4458baec32c0bfe565a9 [term 1 LEADER]: Becoming Leader. State: Replica: dfe3c2216efc4458baec32c0bfe565a9, State: Running, Role: LEADER
I20260928 06:43:52.164740 5878 consensus_queue.cc:237] T f1e77641d7ee46d9a9badafead0c10f2 P dfe3c2216efc4458baec32c0bfe565a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfe3c2216efc4458baec32c0bfe565a9" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 39587 } } peers { permanent_uuid: "5ec43ba041ec4f5fa04788eea02b90ad" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 37919 } } peers { permanent_uuid: "2b61025b43494dc8996fe1d6d32ebb15" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40491 } }
I20260928 06:43:52.165567 5405 catalog_manager.cc:5495] T f1e77641d7ee46d9a9badafead0c10f2 P dfe3c2216efc4458baec32c0bfe565a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to dfe3c2216efc4458baec32c0bfe565a9 (127.18.211.66). New cstate: current_term: 1 leader_uuid: "dfe3c2216efc4458baec32c0bfe565a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfe3c2216efc4458baec32c0bfe565a9" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 39587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ec43ba041ec4f5fa04788eea02b90ad" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 37919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b61025b43494dc8996fe1d6d32ebb15" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40491 } health_report { overall_health: UNKNOWN } } }
I20260928 06:43:52.171774 5869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:52.169353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50786 (local address 127.18.211.66:39587)
0928 06:43:52.169402 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:52.169404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:52.169411 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:43:52.169471 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:52.169472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:52.169490 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:52.169547 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:43:52.169674 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:52.170127 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:52.170865 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:52.171053 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:52.171125 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:52.171388 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:43:52.171533 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:43:52.171614 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:52.171667 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 06:43:52.171677 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 06:43:52.173540 5549 raft_consensus.cc:1261] T f1e77641d7ee46d9a9badafead0c10f2 P 5ec43ba041ec4f5fa04788eea02b90ad [term 1 FOLLOWER]: Refusing update from remote peer dfe3c2216efc4458baec32c0bfe565a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:43:52.173542 5811 raft_consensus.cc:1261] T f1e77641d7ee46d9a9badafead0c10f2 P 2b61025b43494dc8996fe1d6d32ebb15 [term 1 FOLLOWER]: Refusing update from remote peer dfe3c2216efc4458baec32c0bfe565a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:43:52.173854 5878 consensus_queue.cc:1035] T f1e77641d7ee46d9a9badafead0c10f2 P dfe3c2216efc4458baec32c0bfe565a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ec43ba041ec4f5fa04788eea02b90ad" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 37919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:43:52.173991 5882 consensus_queue.cc:1035] T f1e77641d7ee46d9a9badafead0c10f2 P dfe3c2216efc4458baec32c0bfe565a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b61025b43494dc8996fe1d6d32ebb15" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:43:52.176179 5885 mvcc.cc:205] Tried to move back new op lower bound from 7334206800580304896 to 7334206800547409920. Current Snapshot: MvccSnapshot[applied={T|T < 7334206800580304896}]
I20260928 06:43:52.176707 5887 mvcc.cc:205] Tried to move back new op lower bound from 7334206800580304896 to 7334206800547409920. Current Snapshot: MvccSnapshot[applied={T|T < 7334206800580304896}]
I20260928 06:43:52.176854 5886 mvcc.cc:205] Tried to move back new op lower bound from 7334206800580304896 to 7334206800547409920. Current Snapshot: MvccSnapshot[applied={T|T < 7334206800580304896}]
I20260928 06:43:52.178202 5406 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:43:52.178287 5406 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:43:52.178963 5406 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6452cc8afd2439e90fa98188f12a0ac: Sending DeleteTablet for 3 replicas of tablet f1e77641d7ee46d9a9badafead0c10f2
I20260928 06:43:52.179280 5660 tablet_service.cc:1514] Processing DeleteTablet for tablet f1e77641d7ee46d9a9badafead0c10f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:43:52 UTC) from {username='slave'} at 127.0.0.1:50776
I20260928 06:43:52.179335 5528 tablet_service.cc:1514] Processing DeleteTablet for tablet f1e77641d7ee46d9a9badafead0c10f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:43:52 UTC) from {username='slave'} at 127.0.0.1:36022
I20260928 06:43:52.179353 5791 tablet_service.cc:1514] Processing DeleteTablet for tablet f1e77641d7ee46d9a9badafead0c10f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:43:52 UTC) from {username='slave'} at 127.0.0.1:53700
I20260928 06:43:52.179634 5895 tablet_replica.cc:330] T f1e77641d7ee46d9a9badafead0c10f2 P dfe3c2216efc4458baec32c0bfe565a9: stopping tablet replica
I20260928 06:43:52.179747 5895 raft_consensus.cc:2229] T f1e77641d7ee46d9a9badafead0c10f2 P dfe3c2216efc4458baec32c0bfe565a9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:43:52.179836 5895 raft_consensus.cc:2258] T f1e77641d7ee46d9a9badafead0c10f2 P dfe3c2216efc4458baec32c0bfe565a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:43:52.179972 5896 tablet_replica.cc:330] T f1e77641d7ee46d9a9badafead0c10f2 P 2b61025b43494dc8996fe1d6d32ebb15: stopping tablet replica
I20260928 06:43:52.180066 5896 raft_consensus.cc:2229] T f1e77641d7ee46d9a9badafead0c10f2 P 2b61025b43494dc8996fe1d6d32ebb15 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:43:52.180064 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 5466
I20260928 06:43:52.180161 5896 raft_consensus.cc:2258] T f1e77641d7ee46d9a9badafead0c10f2 P 2b61025b43494dc8996fe1d6d32ebb15 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:43:52.182243 5896 ts_tablet_manager.cc:1905] T f1e77641d7ee46d9a9badafead0c10f2 P 2b61025b43494dc8996fe1d6d32ebb15: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:43:52.182248 5895 ts_tablet_manager.cc:1905] T f1e77641d7ee46d9a9badafead0c10f2 P dfe3c2216efc4458baec32c0bfe565a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:43:52.183838 5895 ts_tablet_manager.cc:1918] T f1e77641d7ee46d9a9badafead0c10f2 P dfe3c2216efc4458baec32c0bfe565a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:43:52.183910 5895 log.cc:1196] T f1e77641d7ee46d9a9badafead0c10f2 P dfe3c2216efc4458baec32c0bfe565a9: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-1/wal/wals/f1e77641d7ee46d9a9badafead0c10f2
I20260928 06:43:52.184049 5896 ts_tablet_manager.cc:1918] T f1e77641d7ee46d9a9badafead0c10f2 P 2b61025b43494dc8996fe1d6d32ebb15: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:43:52.184131 5895 ts_tablet_manager.cc:1939] T f1e77641d7ee46d9a9badafead0c10f2 P dfe3c2216efc4458baec32c0bfe565a9: Deleting consensus metadata
I20260928 06:43:52.184216 5896 log.cc:1196] T f1e77641d7ee46d9a9badafead0c10f2 P 2b61025b43494dc8996fe1d6d32ebb15: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790577602522428-19277-0/minicluster-data/ts-2/wal/wals/f1e77641d7ee46d9a9badafead0c10f2
I20260928 06:43:52.184509 5896 ts_tablet_manager.cc:1939] T f1e77641d7ee46d9a9badafead0c10f2 P 2b61025b43494dc8996fe1d6d32ebb15: Deleting consensus metadata
I20260928 06:43:52.184556 5392 catalog_manager.cc:4841] TS dfe3c2216efc4458baec32c0bfe565a9 (127.18.211.66:39587): tablet f1e77641d7ee46d9a9badafead0c10f2 (table test-table [id=bae64236b6004fbe9d4f3509443496d7]) successfully deleted
I20260928 06:43:52.185003 5390 catalog_manager.cc:4841] TS 2b61025b43494dc8996fe1d6d32ebb15 (127.18.211.67:40491): tablet f1e77641d7ee46d9a9badafead0c10f2 (table test-table [id=bae64236b6004fbe9d4f3509443496d7]) successfully deleted
W20260928 06:43:52.186820 5389 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.211.65:37919 (error 108)
I20260928 06:43:52.187312 5868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:52.187175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:37919 (local address 127.0.0.1:36026)
0928 06:43:52.187232 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:52.187278 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.65:37919: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 06:43:52.187319 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 5598
W20260928 06:43:52.188807 5389 catalog_manager.cc:4584] TS 5ec43ba041ec4f5fa04788eea02b90ad (127.18.211.65:37919): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f1e77641d7ee46d9a9badafead0c10f2: Network error: Client connection negotiation failed: client connection to 127.18.211.65:37919: connect: Connection refused (error 111)
I20260928 06:43:52.192270 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 5729
I20260928 06:43:52.196417 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 5374
2026-09-28T06:43:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (665 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:43:52 dist-test-slave-b28r krb5kdc[5901](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:43:52 dist-test-slave-b28r krb5kdc[5901](info): set up 2 sockets
Sep 28 06:43:52 dist-test-slave-b28r krb5kdc[5901](info): commencing operation
krb5kdc: starting...
W20260928 06:43:53.147851 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:43:53 dist-test-slave-b28r krb5kdc[5901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:43:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:43:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.211.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
I20260928 06:43:53.259752 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:35357
--webserver_interface=127.18.211.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:42587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:35357
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:43:53.341805 5914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:43:53.342000 5914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:43:53.342021 5914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:43:53.343130 5914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:43:53.343173 5914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:43:53.343187 5914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:43:53.343199 5914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:43:53.343223 5914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:43:53.344640 5914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:42587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:35357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:35357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:43:53.344820 5914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:43:53.344986 5914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:43:53.349462 5922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:53.349434 5919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:53.349444 5920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:53.349702 5914 server_base.cc:1034] running on GCE node
I20260928 06:43:53.349905 5914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:43:53.350186 5914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:43:53.351325 5914 hybrid_clock.cc:648] HybridClock initialized: now 1790577833351304 us; error 29 us; skew 500 ppm
Sep 28 06:43:53 dist-test-slave-b28r krb5kdc[5901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577833, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:43:53.353668 5914 init.cc:377] Logged in from keytab as kudu/127.18.211.126@KRBTEST.COM (short username kudu)
I20260928 06:43:53.354108 5914 webserver.cc:460] Webserver started at http://127.18.211.126:45823/ using document root <none> and password file <none>
I20260928 06:43:53.354327 5914 fs_manager.cc:359] Metadata directory not provided
I20260928 06:43:53.354372 5914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:43:53.354492 5914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:43:53.355579 5914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "19d8116595fc4cf8bd81fce203165475"
format_stamp: "Formatted at 2026-09-28 06:43:53 on dist-test-slave-b28r"
server_key: "6631e77155370d7bdea39f39079aaa81"
server_key_iv: "52d3bd598708b13a05ec679ee1fd6de2"
server_key_version: "encryptionkey@0"
I20260928 06:43:53.355908 5914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "19d8116595fc4cf8bd81fce203165475"
format_stamp: "Formatted at 2026-09-28 06:43:53 on dist-test-slave-b28r"
server_key: "6631e77155370d7bdea39f39079aaa81"
server_key_iv: "52d3bd598708b13a05ec679ee1fd6de2"
server_key_version: "encryptionkey@0"
I20260928 06:43:53.357133 5914 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:43:53.357797 5929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:53.357962 5914 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:43:53.358047 5914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "19d8116595fc4cf8bd81fce203165475"
format_stamp: "Formatted at 2026-09-28 06:43:53 on dist-test-slave-b28r"
server_key: "6631e77155370d7bdea39f39079aaa81"
server_key_iv: "52d3bd598708b13a05ec679ee1fd6de2"
server_key_version: "encryptionkey@0"
I20260928 06:43:53.358100 5914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:43:53.372382 5914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:43:53.374080 5914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:43:53.374187 5914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:43:53.377827 5914 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:35357
I20260928 06:43:53.377871 5981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:35357 every 8 connection(s)
I20260928 06:43:53.378173 5914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:43:53.379032 5982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:43:53.381263 5982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19d8116595fc4cf8bd81fce203165475: Bootstrap starting.
I20260928 06:43:53.381807 5982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19d8116595fc4cf8bd81fce203165475: Neither blocks nor log segments found. Creating new log.
I20260928 06:43:53.382061 5982 log.cc:824] T 00000000000000000000000000000000 P 19d8116595fc4cf8bd81fce203165475: Log is configured to *not* fsync() on all Append() calls
I20260928 06:43:53.382755 5982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19d8116595fc4cf8bd81fce203165475: No bootstrap required, opened a new log
I20260928 06:43:53.384265 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 5914
I20260928 06:43:53.384367 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:43:53.384402 5982 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19d8116595fc4cf8bd81fce203165475 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d8116595fc4cf8bd81fce203165475" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35357 } }
I20260928 06:43:53.384534 19277 external_mini_cluster.cc:1428] Setting key 4c1bcd5b7f1d2751f489b5132db080ab
I20260928 06:43:53.384542 5982 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19d8116595fc4cf8bd81fce203165475 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:43:53.384575 5982 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19d8116595fc4cf8bd81fce203165475 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19d8116595fc4cf8bd81fce203165475, State: Initialized, Role: FOLLOWER
I20260928 06:43:53.384729 5982 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19d8116595fc4cf8bd81fce203165475 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d8116595fc4cf8bd81fce203165475" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35357 } }
I20260928 06:43:53.384815 5982 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19d8116595fc4cf8bd81fce203165475 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:43:53.384855 5982 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19d8116595fc4cf8bd81fce203165475 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:43:53.385095 5982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19d8116595fc4cf8bd81fce203165475 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:43:53.386080 5982 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19d8116595fc4cf8bd81fce203165475 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d8116595fc4cf8bd81fce203165475" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35357 } }
I20260928 06:43:53.386185 5982 leader_election.cc:302] T 00000000000000000000000000000000 P 19d8116595fc4cf8bd81fce203165475 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19d8116595fc4cf8bd81fce203165475; no voters:
I20260928 06:43:53.386344 5982 leader_election.cc:288] T 00000000000000000000000000000000 P 19d8116595fc4cf8bd81fce203165475 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:43:53.386380 5987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19d8116595fc4cf8bd81fce203165475 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:43:53.386516 5987 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19d8116595fc4cf8bd81fce203165475 [term 1 LEADER]: Becoming Leader. State: Replica: 19d8116595fc4cf8bd81fce203165475, State: Running, Role: LEADER
I20260928 06:43:53.386575 5982 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19d8116595fc4cf8bd81fce203165475 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:43:53.386605 5987 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19d8116595fc4cf8bd81fce203165475 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d8116595fc4cf8bd81fce203165475" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35357 } }
I20260928 06:43:53.386953 5989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19d8116595fc4cf8bd81fce203165475 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19d8116595fc4cf8bd81fce203165475" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d8116595fc4cf8bd81fce203165475" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35357 } } }
I20260928 06:43:53.387048 5989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19d8116595fc4cf8bd81fce203165475 [sys.catalog]: This master's current role is: LEADER
I20260928 06:43:53.386942 5987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19d8116595fc4cf8bd81fce203165475 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19d8116595fc4cf8bd81fce203165475. Latest consensus state: current_term: 1 leader_uuid: "19d8116595fc4cf8bd81fce203165475" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d8116595fc4cf8bd81fce203165475" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35357 } } }
I20260928 06:43:53.387257 5987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19d8116595fc4cf8bd81fce203165475 [sys.catalog]: This master's current role is: LEADER
W20260928 06:43:53.388310 6003 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 19d8116595fc4cf8bd81fce203165475: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:43:53.388525 6003 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:43:53.388592 5995 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:43:53.388751 5995 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:43:53.390352 5995 catalog_manager.cc:1347] Generated new cluster ID: 6dbbbea139f84e11a4eb6d9b274088dd
I20260928 06:43:53.390401 5995 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:43:53 dist-test-slave-b28r krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:43:53.395121 5986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:53.384968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44894 (local address 127.18.211.126:35357)
0928 06:43:53.385130 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:53.385139 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:53.385177 (+ 38us) server_negotiation.cc:407] Connection header received
0928 06:43:53.385311 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:53.385319 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:53.385373 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:53.385522 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:53.386071 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:53.387451 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:53.388333 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:53.388921 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:53.389028 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:53.391699 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:53.391703 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:53.391704 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:43:53.391715 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:53.393580 (+ 1865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:53.393931 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:53.393934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:53.393935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:53.394065 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:53.394232 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:53.394233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:53.394233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:53.394528 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:53.394690 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:53.394760 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 06:43:53.394895 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260928 06:43:53.397914 5995 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:43:53.398373 5995 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:43:53.404803 5995 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19d8116595fc4cf8bd81fce203165475: Generated new TSK 0
I20260928 06:43:53.404976 5995 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.211.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab.
Entry for principal kudu/127.18.211.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab.
I20260928 06:43:53.431875 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.65:0
--local_ip_for_outbound_sockets=127.18.211.65
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35357
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:42587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:43:53.510569 6008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:43:53.510772 6008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:43:53.510805 6008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:43:53.511935 6008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:43:53.511972 6008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:43:53.512006 6008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.65
I20260928 06:43:53.513386 6008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:42587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.65.keytab
--principal=kudu/127.18.211.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.65
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:43:53.513604 6008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:43:53.513811 6008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:43:53.518455 6014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:53.518537 6013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:53.518638 6008 server_base.cc:1034] running on GCE node
W20260928 06:43:53.518453 6016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:53.518874 6008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:43:53.519124 6008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:43:53.520296 6008 hybrid_clock.cc:648] HybridClock initialized: now 1790577833520277 us; error 29 us; skew 500 ppm
Sep 28 06:43:53 dist-test-slave-b28r krb5kdc[5901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577833, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:43:53.522584 6008 init.cc:377] Logged in from keytab as kudu/127.18.211.65@KRBTEST.COM (short username kudu)
I20260928 06:43:53.522960 6008 webserver.cc:460] Webserver started at http://127.18.211.65:42149/ using document root <none> and password file <none>
I20260928 06:43:53.523173 6008 fs_manager.cc:359] Metadata directory not provided
I20260928 06:43:53.523217 6008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:43:53.523316 6008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:43:53.524282 6008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-0/data/instance:
uuid: "dc57e29ae8314d9e96463a1b56b5b2e2"
format_stamp: "Formatted at 2026-09-28 06:43:53 on dist-test-slave-b28r"
server_key: "dd04d9ea7dc08765f92f02b6c8fd2c95"
server_key_iv: "2a528b0a2205da5f2b7c421ca8b4e526"
server_key_version: "encryptionkey@0"
I20260928 06:43:53.524571 6008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance:
uuid: "dc57e29ae8314d9e96463a1b56b5b2e2"
format_stamp: "Formatted at 2026-09-28 06:43:53 on dist-test-slave-b28r"
server_key: "dd04d9ea7dc08765f92f02b6c8fd2c95"
server_key_iv: "2a528b0a2205da5f2b7c421ca8b4e526"
server_key_version: "encryptionkey@0"
I20260928 06:43:53.525707 6008 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260928 06:43:53.526373 6023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:53.526500 6008 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:43:53.526577 6008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-0/wal
uuid: "dc57e29ae8314d9e96463a1b56b5b2e2"
format_stamp: "Formatted at 2026-09-28 06:43:53 on dist-test-slave-b28r"
server_key: "dd04d9ea7dc08765f92f02b6c8fd2c95"
server_key_iv: "2a528b0a2205da5f2b7c421ca8b4e526"
server_key_version: "encryptionkey@0"
I20260928 06:43:53.526640 6008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:43:53.545135 6008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:43:53.547207 6008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:43:53.547318 6008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:43:53.547490 6008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:43:53.547730 6008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:43:53.547760 6008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:53.547778 6008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:43:53.547796 6008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:53.553085 6008 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.65:43165
I20260928 06:43:53.553153 6136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.65:43165 every 8 connection(s)
I20260928 06:43:53.553431 6008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
I20260928 06:43:53.556517 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 6008
I20260928 06:43:53.556620 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance
I20260928 06:43:53.556806 19277 external_mini_cluster.cc:1428] Setting key f72ef3c057eaad4fd305289ce2d706bf
Sep 28 06:43:53 dist-test-slave-b28r krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577833, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.65@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:43:53.562973 5986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:53.554161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:34025 (local address 127.18.211.126:35357)
0928 06:43:53.554220 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:53.554222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:53.554397 (+ 175us) server_negotiation.cc:407] Connection header received
0928 06:43:53.554893 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:53.554894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:53.554908 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:53.554948 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:53.555827 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:53.556335 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:53.557522 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:53.557736 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:53.557791 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:53.560103 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:53.560105 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:53.560105 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:43:53.560112 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:53.561694 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:53.562069 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:53.562071 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:53.562071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:53.562134 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:53.562359 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:53.562361 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:53.562361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:53.562496 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:53.562566 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:53.562795 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 06:43:53.562881 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:43:53.563119 6139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:53.554124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35357 (local address 127.18.211.65:34025)
0928 06:43:53.554274 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:53.554315 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:53.554796 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:53.555009 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:53.555012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:53.555570 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:43:53.555755 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:53.555760 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:53.556435 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:53.556436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:53.557442 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:53.557446 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:53.557807 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:53.557809 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:53.557941 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:53.558408 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:43:53.558416 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:43:53.560008 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:43:53.561772 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:43:53.561780 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:43:53.561781 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:43:53.561997 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:43:53.562206 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:43:53.562207 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:43:53.562207 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:43:53.562289 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:43:53.562560 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:43:53.562564 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:43:53.562674 (+ 110us) client_negotiation.cc:769] Sending connection context
0928 06:43:53.562754 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 06:43:53.562876 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
I20260928 06:43:53.563632 6137 heartbeater.cc:347] Connected to a master server at 127.18.211.126:35357
I20260928 06:43:53.563741 6137 heartbeater.cc:464] Registering TS with master...
I20260928 06:43:53.563922 6137 heartbeater.cc:511] Master 127.18.211.126:35357 requested a full tablet report, sending...
I20260928 06:43:53.564502 5946 ts_manager.cc:194] Registered new tserver with Master: dc57e29ae8314d9e96463a1b56b5b2e2 (127.18.211.65:43165)
WARNING: no policy specified for kudu/127.18.211.66@KRBTEST.COM; defaulting to no policy
I20260928 06:43:53.565428 5946 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.65@KRBTEST.COM'} at 127.18.211.65:34025
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab.
Entry for principal kudu/127.18.211.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab.
I20260928 06:43:53.579440 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.66:0
--local_ip_for_outbound_sockets=127.18.211.66
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35357
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:42587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:43:53.659278 6142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:43:53.659485 6142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:43:53.659508 6142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:43:53.660673 6142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:43:53.660714 6142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:43:53.660740 6142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.66
I20260928 06:43:53.662177 6142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:42587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.66.keytab
--principal=kudu/127.18.211.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.66
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:43:53.662402 6142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:43:53.662653 6142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:43:53.667335 6147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:53.667348 6150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:53.667526 6148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:53.667531 6142 server_base.cc:1034] running on GCE node
I20260928 06:43:53.667730 6142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:43:53.667963 6142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:43:53.669121 6142 hybrid_clock.cc:648] HybridClock initialized: now 1790577833669104 us; error 27 us; skew 500 ppm
Sep 28 06:43:53 dist-test-slave-b28r krb5kdc[5901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577833, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:43:53.671474 6142 init.cc:377] Logged in from keytab as kudu/127.18.211.66@KRBTEST.COM (short username kudu)
I20260928 06:43:53.671871 6142 webserver.cc:460] Webserver started at http://127.18.211.66:41369/ using document root <none> and password file <none>
I20260928 06:43:53.672053 6142 fs_manager.cc:359] Metadata directory not provided
I20260928 06:43:53.672099 6142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:43:53.672226 6142 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:43:53.673244 6142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-1/data/instance:
uuid: "39c464dadaaa44d293120d2ec89a0132"
format_stamp: "Formatted at 2026-09-28 06:43:53 on dist-test-slave-b28r"
server_key: "9af080fed9b7548b019b68546900ca7b"
server_key_iv: "4ff5fb5b847945d0eacfef588fc13158"
server_key_version: "encryptionkey@0"
I20260928 06:43:53.673563 6142 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance:
uuid: "39c464dadaaa44d293120d2ec89a0132"
format_stamp: "Formatted at 2026-09-28 06:43:53 on dist-test-slave-b28r"
server_key: "9af080fed9b7548b019b68546900ca7b"
server_key_iv: "4ff5fb5b847945d0eacfef588fc13158"
server_key_version: "encryptionkey@0"
I20260928 06:43:53.674701 6142 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:43:53.675318 6157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:53.675462 6142 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:43:53.675534 6142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-1/wal
uuid: "39c464dadaaa44d293120d2ec89a0132"
format_stamp: "Formatted at 2026-09-28 06:43:53 on dist-test-slave-b28r"
server_key: "9af080fed9b7548b019b68546900ca7b"
server_key_iv: "4ff5fb5b847945d0eacfef588fc13158"
server_key_version: "encryptionkey@0"
I20260928 06:43:53.675594 6142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:43:53.690850 6142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:43:53.692914 6142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:43:53.693032 6142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:43:53.693250 6142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:43:53.693547 6142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:43:53.693578 6142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:53.693602 6142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:43:53.693625 6142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:53.699222 6142 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.66:39077
I20260928 06:43:53.699275 6270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.66:39077 every 8 connection(s)
I20260928 06:43:53.699560 6142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
I20260928 06:43:53.703943 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 6142
I20260928 06:43:53.704070 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance
I20260928 06:43:53.704289 19277 external_mini_cluster.cc:1428] Setting key b0daaad4f39d7ea12bb1427e432ae051
Sep 28 06:43:53 dist-test-slave-b28r krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577833, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.66@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:43:53.708799 5986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:53.700237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:39137 (local address 127.18.211.126:35357)
0928 06:43:53.700356 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:53.700358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:53.700546 (+ 188us) server_negotiation.cc:407] Connection header received
0928 06:43:53.701095 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:53.701096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:53.701109 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:53.701164 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:53.702089 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:53.702595 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:53.703589 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:53.703788 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:53.703844 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:53.706052 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:53.706054 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:53.706054 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:43:53.706062 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:53.707428 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:53.707864 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:53.707864 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:53.707865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:53.707930 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:53.708172 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:53.708172 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:53.708173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:53.708283 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:53.708338 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:53.708664 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 06:43:53.708736 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 06:43:53.709025 6273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:53.700229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35357 (local address 127.18.211.66:39137)
0928 06:43:53.700420 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:53.700462 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:53.700988 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:53.701238 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:53.701243 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:53.701800 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:43:53.701990 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:53.701995 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:53.702679 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:53.702680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:53.703501 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:53.703504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:53.703857 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:53.703858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:53.703976 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:53.704433 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:43:53.704444 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:43:53.705968 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:43:53.707508 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:43:53.707516 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:43:53.707518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:43:53.707791 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:43:53.707980 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:43:53.707981 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:43:53.707981 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:43:53.708108 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:43:53.708350 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:43:53.708357 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:43:53.708524 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 06:43:53.708643 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 06:43:53.708806 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":83,"threads_started":1}
I20260928 06:43:53.709561 6271 heartbeater.cc:347] Connected to a master server at 127.18.211.126:35357
I20260928 06:43:53.709673 6271 heartbeater.cc:464] Registering TS with master...
I20260928 06:43:53.709861 6271 heartbeater.cc:511] Master 127.18.211.126:35357 requested a full tablet report, sending...
I20260928 06:43:53.710242 5946 ts_manager.cc:194] Registered new tserver with Master: 39c464dadaaa44d293120d2ec89a0132 (127.18.211.66:39077)
I20260928 06:43:53.710608 5946 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.66@KRBTEST.COM'} at 127.18.211.66:39137
WARNING: no policy specified for kudu/127.18.211.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab.
Entry for principal kudu/127.18.211.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab.
I20260928 06:43:53.727857 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.67:0
--local_ip_for_outbound_sockets=127.18.211.67
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35357
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:42587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:43:53.807116 6276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:43:53.807315 6276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:43:53.807344 6276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:43:53.808468 6276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:43:53.808507 6276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:43:53.808540 6276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.67
I20260928 06:43:53.809942 6276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:42587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.67.keytab
--principal=kudu/127.18.211.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:35357
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.67
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:43:53.810163 6276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:43:53.810393 6276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:43:53.815030 6281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:53.815188 6284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:53.815056 6282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:53.815450 6276 server_base.cc:1034] running on GCE node
I20260928 06:43:53.815636 6276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:43:53.815850 6276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:43:53.816987 6276 hybrid_clock.cc:648] HybridClock initialized: now 1790577833816974 us; error 32 us; skew 500 ppm
Sep 28 06:43:53 dist-test-slave-b28r krb5kdc[5901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577833, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:43:53.819362 6276 init.cc:377] Logged in from keytab as kudu/127.18.211.67@KRBTEST.COM (short username kudu)
I20260928 06:43:53.819806 6276 webserver.cc:460] Webserver started at http://127.18.211.67:46371/ using document root <none> and password file <none>
I20260928 06:43:53.820004 6276 fs_manager.cc:359] Metadata directory not provided
I20260928 06:43:53.820051 6276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:43:53.820183 6276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:43:53.821190 6276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-2/data/instance:
uuid: "8ce5883dcb084078b5d9cf762a9ca741"
format_stamp: "Formatted at 2026-09-28 06:43:53 on dist-test-slave-b28r"
server_key: "49c773fd0ad3a22a24009f278b722b98"
server_key_iv: "22b0619ff01eea649ad8b202d0dcc606"
server_key_version: "encryptionkey@0"
I20260928 06:43:53.821513 6276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance:
uuid: "8ce5883dcb084078b5d9cf762a9ca741"
format_stamp: "Formatted at 2026-09-28 06:43:53 on dist-test-slave-b28r"
server_key: "49c773fd0ad3a22a24009f278b722b98"
server_key_iv: "22b0619ff01eea649ad8b202d0dcc606"
server_key_version: "encryptionkey@0"
I20260928 06:43:53.822695 6276 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:43:53.823343 6291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:53.823490 6276 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:43:53.823570 6276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-2/wal
uuid: "8ce5883dcb084078b5d9cf762a9ca741"
format_stamp: "Formatted at 2026-09-28 06:43:53 on dist-test-slave-b28r"
server_key: "49c773fd0ad3a22a24009f278b722b98"
server_key_iv: "22b0619ff01eea649ad8b202d0dcc606"
server_key_version: "encryptionkey@0"
I20260928 06:43:53.823627 6276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:43:53.844130 6276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:43:53.846275 6276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:43:53.846421 6276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:43:53.846652 6276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:43:53.846969 6276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:43:53.847003 6276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:53.847031 6276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:43:53.847048 6276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:53.852654 6276 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.67:40879
I20260928 06:43:53.852705 6404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.67:40879 every 8 connection(s)
I20260928 06:43:53.853004 6276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
Sep 28 06:43:53 dist-test-slave-b28r krb5kdc[5901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577833, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.67@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:43:53.862582 5986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:53.853790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:40563 (local address 127.18.211.126:35357)
0928 06:43:53.853862 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:53.853864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:53.854049 (+ 185us) server_negotiation.cc:407] Connection header received
0928 06:43:53.854603 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:53.854604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:53.854619 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:53.854691 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:53.855775 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:53.856276 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:53.857336 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:53.857528 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:53.857593 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:53.859870 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:53.859871 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:53.859872 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:43:53.859880 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:53.861310 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:53.861681 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:53.861683 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:53.861683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:53.861741 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:53.861982 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:53.861983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:53.861984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:53.862096 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:53.862154 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:53.862415 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 06:43:53.862493 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 06:43:53.862767 6407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:53.853683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:35357 (local address 127.18.211.67:40563)
0928 06:43:53.853909 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:53.853957 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:53.854492 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:53.854777 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:53.854782 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:53.855450 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:43:53.855662 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:53.855668 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:53.856373 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:53.856374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:53.857239 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:53.857242 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:53.857607 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:53.857608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:53.857765 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:53.858294 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:43:53.858303 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:43:53.859770 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:43:53.861392 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:43:53.861399 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:43:53.861401 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:43:53.861615 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:43:53.861795 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:43:53.861796 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:43:53.861796 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:43:53.861930 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:43:53.862153 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:43:53.862159 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:43:53.862295 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 06:43:53.862383 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 06:43:53.862537 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":78,"threads_started":1}
I20260928 06:43:53.863019 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 6276
I20260928 06:43:53.863140 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance
I20260928 06:43:53.863337 19277 external_mini_cluster.cc:1428] Setting key 63ed59d720f988000e2ab50da15801b2
I20260928 06:43:53.863503 6405 heartbeater.cc:347] Connected to a master server at 127.18.211.126:35357
I20260928 06:43:53.863587 6405 heartbeater.cc:464] Registering TS with master...
I20260928 06:43:53.863777 6405 heartbeater.cc:511] Master 127.18.211.126:35357 requested a full tablet report, sending...
I20260928 06:43:53.864176 5945 ts_manager.cc:194] Registered new tserver with Master: 8ce5883dcb084078b5d9cf762a9ca741 (127.18.211.67:40879)
I20260928 06:43:53.864534 5945 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.211.67@KRBTEST.COM'} at 127.18.211.67:40563
I20260928 06:43:53.875069 19277 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:43:53.882476 5986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:53.876457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44896 (local address 127.18.211.126:35357)
0928 06:43:53.876536 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:53.876537 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:53.876605 (+ 68us) server_negotiation.cc:407] Connection header received
0928 06:43:53.876682 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:53.876683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:53.876695 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:53.876752 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:53.877393 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:53.877892 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:53.878678 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:53.878870 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:53.878926 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:53.880165 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:53.880167 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:53.880167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:43:53.880175 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:53.881255 (+ 1080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:53.881615 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:53.881617 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:53.881617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:53.881697 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:53.881895 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:53.881895 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:53.881896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:53.882072 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:53.882168 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:53.882301 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 06:43:53.882380 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 06:43:53.884059 5945 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:43:53.885696 5945 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:43:53.892504 6415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:53.888154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:43165 (local address 127.0.0.1:40364)
0928 06:43:53.888692 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:53.888702 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:53.888758 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:53.888969 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:53.888970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:53.888972 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:43:53.889016 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:53.889018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:53.890288 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:53.890289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:53.891351 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:53.891353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:53.892232 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:53.892233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:53.892308 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:53.892323 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 06:43:53.892369 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 06:43:53.892419 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":55,"threads_started":1}
I20260928 06:43:53.892815 6417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:53.888194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45010 (local address 127.18.211.66:39077)
0928 06:43:53.888409 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:53.888414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:53.888424 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:43:53.888469 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:53.888471 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:53.888489 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:53.888581 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:43:53.888735 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:53.889871 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:53.891413 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:53.892135 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:53.892208 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:53.892270 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:53.892699 (+ 429us) server_negotiation.cc:299] Negotiation successful
0928 06:43:53.892741 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":93,"threads_started":1}
I20260928 06:43:53.892923 6414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:53.887975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:40879 (local address 127.0.0.1:41164)
0928 06:43:53.888341 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:53.888350 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:53.888395 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:53.889077 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:53.889078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:53.889081 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:43:53.889155 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:53.889157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:53.890520 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:53.890521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:53.891673 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:53.891677 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:53.892688 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:53.892689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:53.892768 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:53.892773 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 06:43:53.892806 (+ 33us) client_negotiation.cc:240] Negotiation successful
0928 06:43:53.892846 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":53,"threads_started":1}
I20260928 06:43:53.893034 6413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:53.887956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:39077 (local address 127.0.0.1:45010)
0928 06:43:53.888058 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:53.888095 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:53.888154 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:53.888586 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:53.888589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:53.888591 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:43:53.888654 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:53.888655 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:53.889984 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:53.889985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:53.891321 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:53.891324 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:53.892299 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:53.892300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:53.892347 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:53.892350 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:43:53.892391 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 06:43:53.892423 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":54,"threads_started":1}
I20260928 06:43:53.893251 6418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:53.888334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40364 (local address 127.18.211.65:43165)
0928 06:43:53.888514 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:53.888519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:53.888762 (+ 243us) server_negotiation.cc:407] Connection header received
0928 06:43:53.888834 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:53.888836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:53.888853 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:53.889088 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:43:53.889163 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:53.890180 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:53.891437 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:53.892135 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:53.892465 (+ 330us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:53.892529 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:53.892791 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 06:43:53.892835 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":113,"threads_started":1}
I20260928 06:43:53.893378 6416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:53.888205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41164 (local address 127.18.211.67:40879)
0928 06:43:53.888898 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:53.888902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:53.888913 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:43:53.888952 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:53.888954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:53.888970 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:53.889239 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:43:53.889314 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:53.890384 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:53.891797 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:53.892553 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:53.893134 (+ 581us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:53.893217 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:53.893247 (+ 30us) server_negotiation.cc:299] Negotiation successful
0928 06:43:53.893294 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":86,"threads_started":1}
I20260928 06:43:53.893806 6205 tablet_service.cc:1467] Processing CreateTablet for tablet dbeb124d9074455c8472be787cf233c8 (DEFAULT_TABLE table=test-table [id=e3dd85fba620477d90b58f485886203f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:43:53.894151 6205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbeb124d9074455c8472be787cf233c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:43:53.894083 6339 tablet_service.cc:1467] Processing CreateTablet for tablet dbeb124d9074455c8472be787cf233c8 (DEFAULT_TABLE table=test-table [id=e3dd85fba620477d90b58f485886203f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:43:53.894451 6339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbeb124d9074455c8472be787cf233c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:43:53.898037 6420 tablet_bootstrap.cc:492] T dbeb124d9074455c8472be787cf233c8 P 8ce5883dcb084078b5d9cf762a9ca741: Bootstrap starting.
I20260928 06:43:53.898171 6071 tablet_service.cc:1467] Processing CreateTablet for tablet dbeb124d9074455c8472be787cf233c8 (DEFAULT_TABLE table=test-table [id=e3dd85fba620477d90b58f485886203f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:43:53.898628 6071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbeb124d9074455c8472be787cf233c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:43:53.899184 6419 tablet_bootstrap.cc:492] T dbeb124d9074455c8472be787cf233c8 P 39c464dadaaa44d293120d2ec89a0132: Bootstrap starting.
I20260928 06:43:53.899731 6419 tablet_bootstrap.cc:654] T dbeb124d9074455c8472be787cf233c8 P 39c464dadaaa44d293120d2ec89a0132: Neither blocks nor log segments found. Creating new log.
I20260928 06:43:53.899767 6420 tablet_bootstrap.cc:654] T dbeb124d9074455c8472be787cf233c8 P 8ce5883dcb084078b5d9cf762a9ca741: Neither blocks nor log segments found. Creating new log.
I20260928 06:43:53.900166 6419 log.cc:824] T dbeb124d9074455c8472be787cf233c8 P 39c464dadaaa44d293120d2ec89a0132: Log is configured to *not* fsync() on all Append() calls
I20260928 06:43:53.900319 6420 log.cc:824] T dbeb124d9074455c8472be787cf233c8 P 8ce5883dcb084078b5d9cf762a9ca741: Log is configured to *not* fsync() on all Append() calls
I20260928 06:43:53.900964 6419 tablet_bootstrap.cc:492] T dbeb124d9074455c8472be787cf233c8 P 39c464dadaaa44d293120d2ec89a0132: No bootstrap required, opened a new log
I20260928 06:43:53.901023 6419 ts_tablet_manager.cc:1397] T dbeb124d9074455c8472be787cf233c8 P 39c464dadaaa44d293120d2ec89a0132: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 06:43:53.901548 6423 tablet_bootstrap.cc:492] T dbeb124d9074455c8472be787cf233c8 P dc57e29ae8314d9e96463a1b56b5b2e2: Bootstrap starting.
I20260928 06:43:53.902210 6420 tablet_bootstrap.cc:492] T dbeb124d9074455c8472be787cf233c8 P 8ce5883dcb084078b5d9cf762a9ca741: No bootstrap required, opened a new log
I20260928 06:43:53.902271 6420 ts_tablet_manager.cc:1397] T dbeb124d9074455c8472be787cf233c8 P 8ce5883dcb084078b5d9cf762a9ca741: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 06:43:53.902557 6419 raft_consensus.cc:348] T dbeb124d9074455c8472be787cf233c8 P 39c464dadaaa44d293120d2ec89a0132 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39c464dadaaa44d293120d2ec89a0132" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 39077 } } peers { permanent_uuid: "8ce5883dcb084078b5d9cf762a9ca741" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40879 } } peers { permanent_uuid: "dc57e29ae8314d9e96463a1b56b5b2e2" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 43165 } }
I20260928 06:43:53.902719 6419 raft_consensus.cc:374] T dbeb124d9074455c8472be787cf233c8 P 39c464dadaaa44d293120d2ec89a0132 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:43:53.902745 6419 raft_consensus.cc:729] T dbeb124d9074455c8472be787cf233c8 P 39c464dadaaa44d293120d2ec89a0132 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39c464dadaaa44d293120d2ec89a0132, State: Initialized, Role: FOLLOWER
I20260928 06:43:53.902865 6419 consensus_queue.cc:260] T dbeb124d9074455c8472be787cf233c8 P 39c464dadaaa44d293120d2ec89a0132 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39c464dadaaa44d293120d2ec89a0132" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 39077 } } peers { permanent_uuid: "8ce5883dcb084078b5d9cf762a9ca741" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40879 } } peers { permanent_uuid: "dc57e29ae8314d9e96463a1b56b5b2e2" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 43165 } }
I20260928 06:43:53.903182 6419 ts_tablet_manager.cc:1428] T dbeb124d9074455c8472be787cf233c8 P 39c464dadaaa44d293120d2ec89a0132: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 06:43:53.903190 6271 heartbeater.cc:503] Master 127.18.211.126:35357 was elected leader, sending a full tablet report...
I20260928 06:43:53.903407 6423 tablet_bootstrap.cc:654] T dbeb124d9074455c8472be787cf233c8 P dc57e29ae8314d9e96463a1b56b5b2e2: Neither blocks nor log segments found. Creating new log.
I20260928 06:43:53.903982 6423 log.cc:824] T dbeb124d9074455c8472be787cf233c8 P dc57e29ae8314d9e96463a1b56b5b2e2: Log is configured to *not* fsync() on all Append() calls
I20260928 06:43:53.903959 6420 raft_consensus.cc:348] T dbeb124d9074455c8472be787cf233c8 P 8ce5883dcb084078b5d9cf762a9ca741 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39c464dadaaa44d293120d2ec89a0132" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 39077 } } peers { permanent_uuid: "8ce5883dcb084078b5d9cf762a9ca741" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40879 } } peers { permanent_uuid: "dc57e29ae8314d9e96463a1b56b5b2e2" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 43165 } }
I20260928 06:43:53.904105 6420 raft_consensus.cc:374] T dbeb124d9074455c8472be787cf233c8 P 8ce5883dcb084078b5d9cf762a9ca741 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:43:53.904155 6420 raft_consensus.cc:729] T dbeb124d9074455c8472be787cf233c8 P 8ce5883dcb084078b5d9cf762a9ca741 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ce5883dcb084078b5d9cf762a9ca741, State: Initialized, Role: FOLLOWER
I20260928 06:43:53.904276 6420 consensus_queue.cc:260] T dbeb124d9074455c8472be787cf233c8 P 8ce5883dcb084078b5d9cf762a9ca741 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39c464dadaaa44d293120d2ec89a0132" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 39077 } } peers { permanent_uuid: "8ce5883dcb084078b5d9cf762a9ca741" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40879 } } peers { permanent_uuid: "dc57e29ae8314d9e96463a1b56b5b2e2" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 43165 } }
I20260928 06:43:53.904551 6420 ts_tablet_manager.cc:1428] T dbeb124d9074455c8472be787cf233c8 P 8ce5883dcb084078b5d9cf762a9ca741: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:43:53.904646 6405 heartbeater.cc:503] Master 127.18.211.126:35357 was elected leader, sending a full tablet report...
I20260928 06:43:53.904780 6423 tablet_bootstrap.cc:492] T dbeb124d9074455c8472be787cf233c8 P dc57e29ae8314d9e96463a1b56b5b2e2: No bootstrap required, opened a new log
I20260928 06:43:53.904850 6423 ts_tablet_manager.cc:1397] T dbeb124d9074455c8472be787cf233c8 P dc57e29ae8314d9e96463a1b56b5b2e2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 06:43:53.906520 6423 raft_consensus.cc:348] T dbeb124d9074455c8472be787cf233c8 P dc57e29ae8314d9e96463a1b56b5b2e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39c464dadaaa44d293120d2ec89a0132" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 39077 } } peers { permanent_uuid: "8ce5883dcb084078b5d9cf762a9ca741" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40879 } } peers { permanent_uuid: "dc57e29ae8314d9e96463a1b56b5b2e2" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 43165 } }
I20260928 06:43:53.906667 6423 raft_consensus.cc:374] T dbeb124d9074455c8472be787cf233c8 P dc57e29ae8314d9e96463a1b56b5b2e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:43:53.906692 6423 raft_consensus.cc:729] T dbeb124d9074455c8472be787cf233c8 P dc57e29ae8314d9e96463a1b56b5b2e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc57e29ae8314d9e96463a1b56b5b2e2, State: Initialized, Role: FOLLOWER
I20260928 06:43:53.906796 6423 consensus_queue.cc:260] T dbeb124d9074455c8472be787cf233c8 P dc57e29ae8314d9e96463a1b56b5b2e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39c464dadaaa44d293120d2ec89a0132" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 39077 } } peers { permanent_uuid: "8ce5883dcb084078b5d9cf762a9ca741" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40879 } } peers { permanent_uuid: "dc57e29ae8314d9e96463a1b56b5b2e2" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 43165 } }
I20260928 06:43:53.907065 6423 ts_tablet_manager.cc:1428] T dbeb124d9074455c8472be787cf233c8 P dc57e29ae8314d9e96463a1b56b5b2e2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:43:53.907105 6137 heartbeater.cc:503] Master 127.18.211.126:35357 was elected leader, sending a full tablet report...
W20260928 06:43:53.950127 6272 tablet.cc:2369] T dbeb124d9074455c8472be787cf233c8 P 39c464dadaaa44d293120d2ec89a0132: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:43:54.054195 6138 tablet.cc:2369] T dbeb124d9074455c8472be787cf233c8 P dc57e29ae8314d9e96463a1b56b5b2e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 06:43:54.103576 6406 tablet.cc:2369] T dbeb124d9074455c8472be787cf233c8 P 8ce5883dcb084078b5d9cf762a9ca741: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:43:54.129701 6424 raft_consensus.cc:482] T dbeb124d9074455c8472be787cf233c8 P 39c464dadaaa44d293120d2ec89a0132 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:43:54.129839 6424 raft_consensus.cc:504] T dbeb124d9074455c8472be787cf233c8 P 39c464dadaaa44d293120d2ec89a0132 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39c464dadaaa44d293120d2ec89a0132" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 39077 } } peers { permanent_uuid: "8ce5883dcb084078b5d9cf762a9ca741" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40879 } } peers { permanent_uuid: "dc57e29ae8314d9e96463a1b56b5b2e2" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 43165 } }
I20260928 06:43:54.130261 6424 leader_election.cc:288] T dbeb124d9074455c8472be787cf233c8 P 39c464dadaaa44d293120d2ec89a0132 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ce5883dcb084078b5d9cf762a9ca741 (127.18.211.67:40879), dc57e29ae8314d9e96463a1b56b5b2e2 (127.18.211.65:43165)
I20260928 06:43:54.134161 6416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:54.130586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:43483 (local address 127.18.211.67:40879)
0928 06:43:54.130639 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:54.130641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:54.130747 (+ 106us) server_negotiation.cc:407] Connection header received
0928 06:43:54.130776 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:54.130777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:54.130785 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:54.130826 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:43:54.130959 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.131679 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.133175 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.133755 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.133820 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:54.133830 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:54.134044 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 06:43:54.134088 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 06:43:54.134162 6418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:54.130546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:58485 (local address 127.18.211.65:43165)
0928 06:43:54.130592 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:54.130594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:54.130605 (+ 11us) server_negotiation.cc:407] Connection header received
0928 06:43:54.130636 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:54.130637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:54.130648 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:54.130714 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:43:54.130883 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.131680 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.133183 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.133742 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.133820 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:54.133830 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:54.134053 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 06:43:54.134102 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:43:54.134308 6273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:54.130447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:43165 (local address 127.18.211.66:58485)
0928 06:43:54.130494 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:54.130506 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:54.130558 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:54.130738 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:54.130739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:54.130740 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:43:54.130816 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:54.130818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.131778 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.131779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:54.133070 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:54.133073 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.133857 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.133858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:54.133960 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:54.133963 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:43:54.134036 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 06:43:54.134086 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":36}
I20260928 06:43:54.134409 6428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:54.130450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:40879 (local address 127.18.211.66:43483)
0928 06:43:54.130669 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:54.130678 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:54.130733 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:54.130842 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:54.130843 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:54.130844 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:43:54.130899 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:54.130900 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.131778 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.131779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:54.133070 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:54.133073 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.133864 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.133865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:54.133955 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:54.133961 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 06:43:54.134036 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 06:43:54.134086 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":83,"thread_start_us":107,"threads_started":1}
I20260928 06:43:54.134779 6091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbeb124d9074455c8472be787cf233c8" candidate_uuid: "39c464dadaaa44d293120d2ec89a0132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc57e29ae8314d9e96463a1b56b5b2e2" is_pre_election: true
I20260928 06:43:54.134797 6359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbeb124d9074455c8472be787cf233c8" candidate_uuid: "39c464dadaaa44d293120d2ec89a0132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ce5883dcb084078b5d9cf762a9ca741" is_pre_election: true
I20260928 06:43:54.134922 6091 raft_consensus.cc:2454] T dbeb124d9074455c8472be787cf233c8 P dc57e29ae8314d9e96463a1b56b5b2e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39c464dadaaa44d293120d2ec89a0132 in term 0.
I20260928 06:43:54.134945 6359 raft_consensus.cc:2454] T dbeb124d9074455c8472be787cf233c8 P 8ce5883dcb084078b5d9cf762a9ca741 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39c464dadaaa44d293120d2ec89a0132 in term 0.
I20260928 06:43:54.135187 6158 leader_election.cc:302] T dbeb124d9074455c8472be787cf233c8 P 39c464dadaaa44d293120d2ec89a0132 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39c464dadaaa44d293120d2ec89a0132, dc57e29ae8314d9e96463a1b56b5b2e2; no voters:
I20260928 06:43:54.135376 6424 raft_consensus.cc:2781] T dbeb124d9074455c8472be787cf233c8 P 39c464dadaaa44d293120d2ec89a0132 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:43:54.135429 6424 raft_consensus.cc:482] T dbeb124d9074455c8472be787cf233c8 P 39c464dadaaa44d293120d2ec89a0132 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:43:54.135444 6424 raft_consensus.cc:3037] T dbeb124d9074455c8472be787cf233c8 P 39c464dadaaa44d293120d2ec89a0132 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:43:54.136180 6424 raft_consensus.cc:504] T dbeb124d9074455c8472be787cf233c8 P 39c464dadaaa44d293120d2ec89a0132 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39c464dadaaa44d293120d2ec89a0132" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 39077 } } peers { permanent_uuid: "8ce5883dcb084078b5d9cf762a9ca741" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40879 } } peers { permanent_uuid: "dc57e29ae8314d9e96463a1b56b5b2e2" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 43165 } }
I20260928 06:43:54.136314 6424 leader_election.cc:288] T dbeb124d9074455c8472be787cf233c8 P 39c464dadaaa44d293120d2ec89a0132 [CANDIDATE]: Term 1 election: Requested vote from peers 8ce5883dcb084078b5d9cf762a9ca741 (127.18.211.67:40879), dc57e29ae8314d9e96463a1b56b5b2e2 (127.18.211.65:43165)
I20260928 06:43:54.136487 6359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbeb124d9074455c8472be787cf233c8" candidate_uuid: "39c464dadaaa44d293120d2ec89a0132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ce5883dcb084078b5d9cf762a9ca741"
I20260928 06:43:54.136529 6091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbeb124d9074455c8472be787cf233c8" candidate_uuid: "39c464dadaaa44d293120d2ec89a0132" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc57e29ae8314d9e96463a1b56b5b2e2"
I20260928 06:43:54.136575 6359 raft_consensus.cc:3037] T dbeb124d9074455c8472be787cf233c8 P 8ce5883dcb084078b5d9cf762a9ca741 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:43:54.136596 6091 raft_consensus.cc:3037] T dbeb124d9074455c8472be787cf233c8 P dc57e29ae8314d9e96463a1b56b5b2e2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:43:54.137487 6091 raft_consensus.cc:2454] T dbeb124d9074455c8472be787cf233c8 P dc57e29ae8314d9e96463a1b56b5b2e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39c464dadaaa44d293120d2ec89a0132 in term 1.
I20260928 06:43:54.137487 6359 raft_consensus.cc:2454] T dbeb124d9074455c8472be787cf233c8 P 8ce5883dcb084078b5d9cf762a9ca741 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39c464dadaaa44d293120d2ec89a0132 in term 1.
I20260928 06:43:54.137722 6161 leader_election.cc:302] T dbeb124d9074455c8472be787cf233c8 P 39c464dadaaa44d293120d2ec89a0132 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39c464dadaaa44d293120d2ec89a0132, 8ce5883dcb084078b5d9cf762a9ca741; no voters:
I20260928 06:43:54.137805 6424 raft_consensus.cc:2781] T dbeb124d9074455c8472be787cf233c8 P 39c464dadaaa44d293120d2ec89a0132 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:43:54.137945 6424 raft_consensus.cc:686] T dbeb124d9074455c8472be787cf233c8 P 39c464dadaaa44d293120d2ec89a0132 [term 1 LEADER]: Becoming Leader. State: Replica: 39c464dadaaa44d293120d2ec89a0132, State: Running, Role: LEADER
I20260928 06:43:54.138036 6424 consensus_queue.cc:237] T dbeb124d9074455c8472be787cf233c8 P 39c464dadaaa44d293120d2ec89a0132 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39c464dadaaa44d293120d2ec89a0132" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 39077 } } peers { permanent_uuid: "8ce5883dcb084078b5d9cf762a9ca741" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40879 } } peers { permanent_uuid: "dc57e29ae8314d9e96463a1b56b5b2e2" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 43165 } }
I20260928 06:43:54.138954 5946 catalog_manager.cc:5495] T dbeb124d9074455c8472be787cf233c8 P 39c464dadaaa44d293120d2ec89a0132 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39c464dadaaa44d293120d2ec89a0132 (127.18.211.66). New cstate: current_term: 1 leader_uuid: "39c464dadaaa44d293120d2ec89a0132" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39c464dadaaa44d293120d2ec89a0132" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 39077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ce5883dcb084078b5d9cf762a9ca741" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc57e29ae8314d9e96463a1b56b5b2e2" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 43165 } health_report { overall_health: UNKNOWN } } }
I20260928 06:43:54.182070 6417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:54.179853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45020 (local address 127.18.211.66:39077)
0928 06:43:54.179910 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:54.179912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:54.179922 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:43:54.179972 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:54.179973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:54.179994 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:54.180058 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:43:54.180172 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.180633 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.181390 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.181579 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.181634 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:54.181776 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:43:54.181879 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:43:54.181953 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:54.181971 (+ 18us) server_negotiation.cc:299] Negotiation successful
0928 06:43:54.181991 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 06:43:54.184103 6091 raft_consensus.cc:1261] T dbeb124d9074455c8472be787cf233c8 P dc57e29ae8314d9e96463a1b56b5b2e2 [term 1 FOLLOWER]: Refusing update from remote peer 39c464dadaaa44d293120d2ec89a0132: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:43:54.184113 6359 raft_consensus.cc:1261] T dbeb124d9074455c8472be787cf233c8 P 8ce5883dcb084078b5d9cf762a9ca741 [term 1 FOLLOWER]: Refusing update from remote peer 39c464dadaaa44d293120d2ec89a0132: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:43:54.184494 6429 consensus_queue.cc:1035] T dbeb124d9074455c8472be787cf233c8 P 39c464dadaaa44d293120d2ec89a0132 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ce5883dcb084078b5d9cf762a9ca741" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 40879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:43:54.184623 6424 consensus_queue.cc:1035] T dbeb124d9074455c8472be787cf233c8 P 39c464dadaaa44d293120d2ec89a0132 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc57e29ae8314d9e96463a1b56b5b2e2" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 43165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:43:54.186934 6432 mvcc.cc:205] Tried to move back new op lower bound from 7334206808815345664 to 7334206808629907456. Current Snapshot: MvccSnapshot[applied={T|T < 7334206808815345664}]
I20260928 06:43:54.187350 6434 mvcc.cc:205] Tried to move back new op lower bound from 7334206808815345664 to 7334206808629907456. Current Snapshot: MvccSnapshot[applied={T|T < 7334206808815345664}]
I20260928 06:43:54.187494 6433 mvcc.cc:205] Tried to move back new op lower bound from 7334206808815345664 to 7334206808629907456. Current Snapshot: MvccSnapshot[applied={T|T < 7334206808815345664}]
I20260928 06:43:54.189007 5945 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:43:54.189108 5945 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:43:54.189838 5945 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 19d8116595fc4cf8bd81fce203165475: Sending DeleteTablet for 3 replicas of tablet dbeb124d9074455c8472be787cf233c8
I20260928 06:43:54.190219 6205 tablet_service.cc:1514] Processing DeleteTablet for tablet dbeb124d9074455c8472be787cf233c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:43:54 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:45010
I20260928 06:43:54.190234 6071 tablet_service.cc:1514] Processing DeleteTablet for tablet dbeb124d9074455c8472be787cf233c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:43:54 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:40364
I20260928 06:43:54.190233 6339 tablet_service.cc:1514] Processing DeleteTablet for tablet dbeb124d9074455c8472be787cf233c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:43:54 UTC) from {username='kudu', principal='kudu/127.18.211.126@KRBTEST.COM'} at 127.0.0.1:41164
I20260928 06:43:54.190431 6443 tablet_replica.cc:330] T dbeb124d9074455c8472be787cf233c8 P dc57e29ae8314d9e96463a1b56b5b2e2: stopping tablet replica
I20260928 06:43:54.190539 6443 raft_consensus.cc:2229] T dbeb124d9074455c8472be787cf233c8 P dc57e29ae8314d9e96463a1b56b5b2e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:43:54.190680 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 6008
I20260928 06:43:54.190703 6443 raft_consensus.cc:2258] T dbeb124d9074455c8472be787cf233c8 P dc57e29ae8314d9e96463a1b56b5b2e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:43:54.191427 6444 tablet_replica.cc:330] T dbeb124d9074455c8472be787cf233c8 P 8ce5883dcb084078b5d9cf762a9ca741: stopping tablet replica
I20260928 06:43:54.191532 6444 raft_consensus.cc:2229] T dbeb124d9074455c8472be787cf233c8 P 8ce5883dcb084078b5d9cf762a9ca741 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:43:54.191633 6444 raft_consensus.cc:2258] T dbeb124d9074455c8472be787cf233c8 P 8ce5883dcb084078b5d9cf762a9ca741 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:43:54.192274 6444 ts_tablet_manager.cc:1905] T dbeb124d9074455c8472be787cf233c8 P 8ce5883dcb084078b5d9cf762a9ca741: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:43:54.193862 6444 ts_tablet_manager.cc:1918] T dbeb124d9074455c8472be787cf233c8 P 8ce5883dcb084078b5d9cf762a9ca741: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:43:54.193935 6444 log.cc:1196] T dbeb124d9074455c8472be787cf233c8 P 8ce5883dcb084078b5d9cf762a9ca741: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790577602522428-19277-0/minicluster-data/ts-2/wal/wals/dbeb124d9074455c8472be787cf233c8
I20260928 06:43:54.194169 6444 ts_tablet_manager.cc:1939] T dbeb124d9074455c8472be787cf233c8 P 8ce5883dcb084078b5d9cf762a9ca741: Deleting consensus metadata
I20260928 06:43:54.194579 5933 catalog_manager.cc:4841] TS 8ce5883dcb084078b5d9cf762a9ca741 (127.18.211.67:40879): tablet dbeb124d9074455c8472be787cf233c8 (table test-table [id=e3dd85fba620477d90b58f485886203f]) successfully deleted
W20260928 06:43:54.196307 5930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.211.65:43165 (error 108)
I20260928 06:43:54.196524 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 6142
I20260928 06:43:54.196538 6442 tablet_replica.cc:330] T dbeb124d9074455c8472be787cf233c8 P 39c464dadaaa44d293120d2ec89a0132: stopping tablet replica
I20260928 06:43:54.196642 6442 raft_consensus.cc:2229] T dbeb124d9074455c8472be787cf233c8 P 39c464dadaaa44d293120d2ec89a0132 [term 1 LEADER]: Raft consensus shutting down.
I20260928 06:43:54.196774 6442 raft_consensus.cc:2258] T dbeb124d9074455c8472be787cf233c8 P 39c464dadaaa44d293120d2ec89a0132 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:43:54.197389 6413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:54.197264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:43165 (local address 127.0.0.1:40378)
0928 06:43:54.197317 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:54.197357 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.65:43165: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 06:43:54.197510 5930 catalog_manager.cc:4584] TS dc57e29ae8314d9e96463a1b56b5b2e2 (127.18.211.65:43165): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dbeb124d9074455c8472be787cf233c8: Network error: Client connection negotiation failed: client connection to 127.18.211.65:43165: connect: Connection refused (error 111)
I20260928 06:43:54.201507 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 6276
I20260928 06:43:54.201546 6413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:54.201449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:39077 (local address 127.0.0.1:45036)
0928 06:43:54.201496 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:54.201520 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.66:39077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 06:43:54.206043 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 5914
2026-09-28T06:43:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2010 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T06:43:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:43:54Z Disabled control of system clock
I20260928 06:43:54.232242 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:33203
--webserver_interface=127.18.211.126
--webserver_port=0
--builtin_ntp_servers=127.18.211.84:38809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:33203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:43:54.310714 6449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:43:54.310925 6449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:43:54.310945 6449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:43:54.312096 6449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:43:54.312137 6449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:43:54.312150 6449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:43:54.312160 6449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:43:54.312171 6449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:43:54.313462 6449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:38809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:33203
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:33203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:43:54.313686 6449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:43:54.313974 6449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:43:54.318518 6457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:54.318524 6454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:54.318563 6455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:54.318807 6449 server_base.cc:1034] running on GCE node
I20260928 06:43:54.319018 6449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:43:54.319347 6449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:43:54.320523 6449 hybrid_clock.cc:648] HybridClock initialized: now 1790577834320502 us; error 32 us; skew 500 ppm
I20260928 06:43:54.321817 6449 webserver.cc:460] Webserver started at http://127.18.211.126:36743/ using document root <none> and password file <none>
I20260928 06:43:54.322032 6449 fs_manager.cc:359] Metadata directory not provided
I20260928 06:43:54.322078 6449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:43:54.322197 6449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:43:54.323297 6449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "878380558fea47fbaf0630c4a4f5870d"
format_stamp: "Formatted at 2026-09-28 06:43:54 on dist-test-slave-b28r"
server_key: "80ad5dab9444886f1ac545d4ec04635f"
server_key_iv: "dc8c4b884edb8cab4e85c4549f2da11c"
server_key_version: "encryptionkey@0"
I20260928 06:43:54.323619 6449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "878380558fea47fbaf0630c4a4f5870d"
format_stamp: "Formatted at 2026-09-28 06:43:54 on dist-test-slave-b28r"
server_key: "80ad5dab9444886f1ac545d4ec04635f"
server_key_iv: "dc8c4b884edb8cab4e85c4549f2da11c"
server_key_version: "encryptionkey@0"
I20260928 06:43:54.324844 6449 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:43:54.325574 6463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:54.325794 6449 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:43:54.325886 6449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "878380558fea47fbaf0630c4a4f5870d"
format_stamp: "Formatted at 2026-09-28 06:43:54 on dist-test-slave-b28r"
server_key: "80ad5dab9444886f1ac545d4ec04635f"
server_key_iv: "dc8c4b884edb8cab4e85c4549f2da11c"
server_key_version: "encryptionkey@0"
I20260928 06:43:54.325958 6449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:43:54.335132 6449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:43:54.335403 6449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:43:54.335525 6449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:43:54.339314 6449 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:33203
I20260928 06:43:54.339356 6515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:33203 every 8 connection(s)
I20260928 06:43:54.339643 6449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:43:54.340562 6516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:43:54.342890 6516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 878380558fea47fbaf0630c4a4f5870d: Bootstrap starting.
I20260928 06:43:54.343479 6516 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 878380558fea47fbaf0630c4a4f5870d: Neither blocks nor log segments found. Creating new log.
I20260928 06:43:54.343739 6516 log.cc:824] T 00000000000000000000000000000000 P 878380558fea47fbaf0630c4a4f5870d: Log is configured to *not* fsync() on all Append() calls
I20260928 06:43:54.344345 6516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 878380558fea47fbaf0630c4a4f5870d: No bootstrap required, opened a new log
I20260928 06:43:54.345952 6516 raft_consensus.cc:348] T 00000000000000000000000000000000 P 878380558fea47fbaf0630c4a4f5870d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "878380558fea47fbaf0630c4a4f5870d" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 33203 } }
I20260928 06:43:54.346141 6516 raft_consensus.cc:374] T 00000000000000000000000000000000 P 878380558fea47fbaf0630c4a4f5870d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:43:54.346134 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 6449
I20260928 06:43:54.346179 6516 raft_consensus.cc:729] T 00000000000000000000000000000000 P 878380558fea47fbaf0630c4a4f5870d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 878380558fea47fbaf0630c4a4f5870d, State: Initialized, Role: FOLLOWER
I20260928 06:43:54.346230 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:43:54.346334 6516 consensus_queue.cc:260] T 00000000000000000000000000000000 P 878380558fea47fbaf0630c4a4f5870d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "878380558fea47fbaf0630c4a4f5870d" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 33203 } }
I20260928 06:43:54.346417 6516 raft_consensus.cc:388] T 00000000000000000000000000000000 P 878380558fea47fbaf0630c4a4f5870d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:43:54.346450 19277 external_mini_cluster.cc:1428] Setting key aa877781be6ea24530ef6ffec62e4975
I20260928 06:43:54.346455 6516 raft_consensus.cc:482] T 00000000000000000000000000000000 P 878380558fea47fbaf0630c4a4f5870d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:43:54.346504 6516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 878380558fea47fbaf0630c4a4f5870d [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:43:54.347174 6516 raft_consensus.cc:504] T 00000000000000000000000000000000 P 878380558fea47fbaf0630c4a4f5870d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "878380558fea47fbaf0630c4a4f5870d" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 33203 } }
I20260928 06:43:54.347293 6516 leader_election.cc:302] T 00000000000000000000000000000000 P 878380558fea47fbaf0630c4a4f5870d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 878380558fea47fbaf0630c4a4f5870d; no voters:
I20260928 06:43:54.347461 6516 leader_election.cc:288] T 00000000000000000000000000000000 P 878380558fea47fbaf0630c4a4f5870d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:43:54.347566 6521 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 878380558fea47fbaf0630c4a4f5870d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:43:54.347733 6516 sys_catalog.cc:564] T 00000000000000000000000000000000 P 878380558fea47fbaf0630c4a4f5870d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:43:54.347757 6521 raft_consensus.cc:686] T 00000000000000000000000000000000 P 878380558fea47fbaf0630c4a4f5870d [term 1 LEADER]: Becoming Leader. State: Replica: 878380558fea47fbaf0630c4a4f5870d, State: Running, Role: LEADER
I20260928 06:43:54.347836 6521 consensus_queue.cc:237] T 00000000000000000000000000000000 P 878380558fea47fbaf0630c4a4f5870d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "878380558fea47fbaf0630c4a4f5870d" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 33203 } }
I20260928 06:43:54.348201 6521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 878380558fea47fbaf0630c4a4f5870d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "878380558fea47fbaf0630c4a4f5870d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "878380558fea47fbaf0630c4a4f5870d" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 33203 } } }
I20260928 06:43:54.348285 6521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 878380558fea47fbaf0630c4a4f5870d [sys.catalog]: This master's current role is: LEADER
I20260928 06:43:54.348407 6521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 878380558fea47fbaf0630c4a4f5870d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 878380558fea47fbaf0630c4a4f5870d. Latest consensus state: current_term: 1 leader_uuid: "878380558fea47fbaf0630c4a4f5870d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "878380558fea47fbaf0630c4a4f5870d" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 33203 } } }
I20260928 06:43:54.348465 6521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 878380558fea47fbaf0630c4a4f5870d [sys.catalog]: This master's current role is: LEADER
I20260928 06:43:54.348763 6529 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:43:54.349273 6529 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:43:54.351123 6529 catalog_manager.cc:1347] Generated new cluster ID: bfcf11da35a04c45b192c8b721fb8244
I20260928 06:43:54.351190 6529 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:43:54.352517 6520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:54.346950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53376 (local address 127.18.211.126:33203)
0928 06:43:54.347166 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:54.347173 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:54.347197 (+ 24us) server_negotiation.cc:407] Connection header received
0928 06:43:54.347321 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:54.347329 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:54.347410 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:54.347532 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:54.347765 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.349504 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.350441 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.350970 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.351115 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:54.351907 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:54.351911 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:54.351911 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:43:54.352072 (+ 161us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:54.352101 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:43:54.352112 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:54.352168 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:54.352244 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 06:43:54.352321 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":85,"threads_started":1}
I20260928 06:43:54.361104 6529 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:43:54.361599 6529 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:43:54.371928 6529 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 878380558fea47fbaf0630c4a4f5870d: Generated new TSK 0
I20260928 06:43:54.372114 6529 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:43:54.379748 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.65:0
--local_ip_for_outbound_sockets=127.18.211.65
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:33203
--builtin_ntp_servers=127.18.211.84:38809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:43:54.461409 6540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:43:54.461616 6540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:43:54.461647 6540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:43:54.462779 6540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:43:54.462821 6540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:43:54.462854 6540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.65
I20260928 06:43:54.464258 6540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:38809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.211.65
--webserver_port=0
--tserver_master_addrs=127.18.211.126:33203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.65
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:43:54.464488 6540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:43:54.464764 6540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:43:54.470501 6545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:54.470501 6546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:54.470569 6548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:54.470760 6540 server_base.cc:1034] running on GCE node
I20260928 06:43:54.471010 6540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:43:54.471338 6540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:43:54.472499 6540 hybrid_clock.cc:648] HybridClock initialized: now 1790577834472476 us; error 38 us; skew 500 ppm
I20260928 06:43:54.473706 6540 webserver.cc:460] Webserver started at http://127.18.211.65:44635/ using document root <none> and password file <none>
I20260928 06:43:54.473924 6540 fs_manager.cc:359] Metadata directory not provided
I20260928 06:43:54.473981 6540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:43:54.474110 6540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:43:54.475214 6540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-0/data/instance:
uuid: "1d216ab3251f49259e63430e1bd6a387"
format_stamp: "Formatted at 2026-09-28 06:43:54 on dist-test-slave-b28r"
server_key: "d38a4567aeeec7b67ea062a4d48a4a0e"
server_key_iv: "cca091124dcf03649ef5df52fa3f864a"
server_key_version: "encryptionkey@0"
I20260928 06:43:54.475538 6540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance:
uuid: "1d216ab3251f49259e63430e1bd6a387"
format_stamp: "Formatted at 2026-09-28 06:43:54 on dist-test-slave-b28r"
server_key: "d38a4567aeeec7b67ea062a4d48a4a0e"
server_key_iv: "cca091124dcf03649ef5df52fa3f864a"
server_key_version: "encryptionkey@0"
I20260928 06:43:54.476725 6540 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:43:54.477419 6554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:54.477586 6540 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:43:54.477658 6540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-0/wal
uuid: "1d216ab3251f49259e63430e1bd6a387"
format_stamp: "Formatted at 2026-09-28 06:43:54 on dist-test-slave-b28r"
server_key: "d38a4567aeeec7b67ea062a4d48a4a0e"
server_key_iv: "cca091124dcf03649ef5df52fa3f864a"
server_key_version: "encryptionkey@0"
I20260928 06:43:54.477726 6540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:43:54.495672 6540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:43:54.495978 6540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:43:54.496107 6540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:43:54.496351 6540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:43:54.496743 6540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:43:54.496779 6540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:54.496803 6540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:43:54.496826 6540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:54.502353 6540 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.65:41679
I20260928 06:43:54.502418 6667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.65:41679 every 8 connection(s)
I20260928 06:43:54.502715 6540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-0/data/info.pb
I20260928 06:43:54.504488 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 6540
I20260928 06:43:54.504585 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-0/wal/instance
I20260928 06:43:54.504768 19277 external_mini_cluster.cc:1428] Setting key f9a06f4d84c4ed9c548a488efea06024
I20260928 06:43:54.505702 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.66:0
--local_ip_for_outbound_sockets=127.18.211.66
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:33203
--builtin_ntp_servers=127.18.211.84:38809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 06:43:54.508255 6520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:54.503483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:38155 (local address 127.18.211.126:33203)
0928 06:43:54.503529 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:54.503530 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:54.503758 (+ 228us) server_negotiation.cc:407] Connection header received
0928 06:43:54.504292 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:54.504294 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:54.504308 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:54.504378 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:54.505151 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.505858 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.506835 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.507030 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.507101 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:54.507992 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:54.507994 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:54.507994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:43:54.507999 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:54.508014 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:43:54.508019 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:54.508064 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:54.508143 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 06:43:54.508176 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:43:54.508451 6670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:54.503449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33203 (local address 127.18.211.65:38155)
0928 06:43:54.503644 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:54.503677 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:54.504170 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:54.504463 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:54.504466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:54.504474 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:43:54.505075 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:54.505082 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.505939 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.505940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:54.506737 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:54.506739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.507120 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.507121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:54.507247 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:54.507847 (+ 600us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:43:54.507848 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:43:54.507887 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:43:54.507911 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:43:54.508070 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:43:54.508074 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:43:54.508077 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:43:54.508139 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 06:43:54.508240 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":91,"threads_started":1}
I20260928 06:43:54.508870 6668 heartbeater.cc:347] Connected to a master server at 127.18.211.126:33203
I20260928 06:43:54.508972 6668 heartbeater.cc:464] Registering TS with master...
I20260928 06:43:54.509148 6668 heartbeater.cc:511] Master 127.18.211.126:33203 requested a full tablet report, sending...
I20260928 06:43:54.509613 6480 ts_manager.cc:194] Registered new tserver with Master: 1d216ab3251f49259e63430e1bd6a387 (127.18.211.65:41679)
I20260928 06:43:54.510190 6480 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.211.65:38155
W20260928 06:43:54.585752 6671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:43:54.585949 6671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:43:54.585978 6671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:43:54.587210 6671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:43:54.587251 6671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:43:54.587283 6671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.66
I20260928 06:43:54.588591 6671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:38809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.211.66
--webserver_port=0
--tserver_master_addrs=127.18.211.126:33203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.66
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:43:54.588809 6671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:43:54.589066 6671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:43:54.593909 6679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:54.593936 6676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:54.593909 6677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:54.593938 6671 server_base.cc:1034] running on GCE node
I20260928 06:43:54.594355 6671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:43:54.594583 6671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:43:54.595729 6671 hybrid_clock.cc:648] HybridClock initialized: now 1790577834595715 us; error 42 us; skew 500 ppm
I20260928 06:43:54.596956 6671 webserver.cc:460] Webserver started at http://127.18.211.66:37549/ using document root <none> and password file <none>
I20260928 06:43:54.597157 6671 fs_manager.cc:359] Metadata directory not provided
I20260928 06:43:54.597199 6671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:43:54.597327 6671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:43:54.598357 6671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-1/data/instance:
uuid: "8f2c5a26fa0b40258c15c4fff9cfa886"
format_stamp: "Formatted at 2026-09-28 06:43:54 on dist-test-slave-b28r"
server_key: "3c454f6ca13adc6c59971750b06a8d3f"
server_key_iv: "d774904e212168c9bfabafc3088f97e6"
server_key_version: "encryptionkey@0"
I20260928 06:43:54.598670 6671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance:
uuid: "8f2c5a26fa0b40258c15c4fff9cfa886"
format_stamp: "Formatted at 2026-09-28 06:43:54 on dist-test-slave-b28r"
server_key: "3c454f6ca13adc6c59971750b06a8d3f"
server_key_iv: "d774904e212168c9bfabafc3088f97e6"
server_key_version: "encryptionkey@0"
I20260928 06:43:54.599931 6671 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:43:54.600626 6685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:54.600768 6671 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:43:54.600839 6671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-1/wal
uuid: "8f2c5a26fa0b40258c15c4fff9cfa886"
format_stamp: "Formatted at 2026-09-28 06:43:54 on dist-test-slave-b28r"
server_key: "3c454f6ca13adc6c59971750b06a8d3f"
server_key_iv: "d774904e212168c9bfabafc3088f97e6"
server_key_version: "encryptionkey@0"
I20260928 06:43:54.600906 6671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:43:54.626417 6671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:43:54.626729 6671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:43:54.626863 6671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:43:54.627115 6671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:43:54.627454 6671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:43:54.627487 6671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:54.627516 6671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:43:54.627542 6671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:54.633215 6671 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.66:36695
I20260928 06:43:54.633283 6798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.66:36695 every 8 connection(s)
I20260928 06:43:54.633571 6671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-1/data/info.pb
I20260928 06:43:54.638589 6520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:54.634368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.66:45335 (local address 127.18.211.126:33203)
0928 06:43:54.634435 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:54.634437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:54.634637 (+ 200us) server_negotiation.cc:407] Connection header received
0928 06:43:54.635175 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:54.635177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:54.635187 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:54.635242 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:54.635886 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.636375 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.637302 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.637491 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.637539 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:54.638313 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:54.638314 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:54.638315 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:43:54.638319 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:54.638334 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:43:54.638337 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:54.638372 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:54.638504 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 06:43:54.638530 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 06:43:54.638835 6801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:54.634286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33203 (local address 127.18.211.66:45335)
0928 06:43:54.634516 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:54.634552 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:54.635065 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:54.635303 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:54.635306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:54.635314 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:43:54.635808 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:54.635813 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.636445 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.636446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:54.637238 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:54.637240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.637583 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.637585 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:54.637706 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:54.638183 (+ 477us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:43:54.638184 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:43:54.638211 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:43:54.638233 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:43:54.638410 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:43:54.638419 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:43:54.638421 (+ 2us) client_negotiation.cc:769] Sending connection context
0928 06:43:54.638503 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 06:43:54.638590 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":98,"threads_started":1}
I20260928 06:43:54.639314 6799 heartbeater.cc:347] Connected to a master server at 127.18.211.126:33203
I20260928 06:43:54.639407 6799 heartbeater.cc:464] Registering TS with master...
I20260928 06:43:54.639592 6799 heartbeater.cc:511] Master 127.18.211.126:33203 requested a full tablet report, sending...
I20260928 06:43:54.639941 6480 ts_manager.cc:194] Registered new tserver with Master: 8f2c5a26fa0b40258c15c4fff9cfa886 (127.18.211.66:36695)
I20260928 06:43:54.640327 6480 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.211.66:45335
I20260928 06:43:54.640609 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 6671
I20260928 06:43:54.640702 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-1/wal/instance
I20260928 06:43:54.640885 19277 external_mini_cluster.cc:1428] Setting key 166f65468b10f64673bd3d7a9a40a715
I20260928 06:43:54.641671 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.211.67:0
--local_ip_for_outbound_sockets=127.18.211.67
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:33203
--builtin_ntp_servers=127.18.211.84:38809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 06:43:54.721256 6802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:43:54.721450 6802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:43:54.721473 6802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:43:54.722626 6802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:43:54.722671 6802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:43:54.722697 6802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.211.67
I20260928 06:43:54.724061 6802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:38809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.211.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.211.67
--webserver_port=0
--tserver_master_addrs=127.18.211.126:33203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.211.67
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:43:54.724292 6802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:43:54.724591 6802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:43:54.729463 6808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:54.729470 6807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:54.729470 6810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:54.729665 6802 server_base.cc:1034] running on GCE node
I20260928 06:43:54.729880 6802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:43:54.730119 6802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:43:54.731302 6802 hybrid_clock.cc:648] HybridClock initialized: now 1790577834731288 us; error 31 us; skew 500 ppm
I20260928 06:43:54.732537 6802 webserver.cc:460] Webserver started at http://127.18.211.67:33649/ using document root <none> and password file <none>
I20260928 06:43:54.732762 6802 fs_manager.cc:359] Metadata directory not provided
I20260928 06:43:54.732812 6802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:43:54.732939 6802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:43:54.733953 6802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-2/data/instance:
uuid: "df5a42072ddb47c7bf556949ea5638ec"
format_stamp: "Formatted at 2026-09-28 06:43:54 on dist-test-slave-b28r"
server_key: "3217f7e8a78955d89b0cfc10824f5a78"
server_key_iv: "2fca1e186bdacd3bca4e53977ae53480"
server_key_version: "encryptionkey@0"
I20260928 06:43:54.734280 6802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance:
uuid: "df5a42072ddb47c7bf556949ea5638ec"
format_stamp: "Formatted at 2026-09-28 06:43:54 on dist-test-slave-b28r"
server_key: "3217f7e8a78955d89b0cfc10824f5a78"
server_key_iv: "2fca1e186bdacd3bca4e53977ae53480"
server_key_version: "encryptionkey@0"
I20260928 06:43:54.735562 6802 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:43:54.736395 6816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:54.736567 6802 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:43:54.736649 6802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-2/wal
uuid: "df5a42072ddb47c7bf556949ea5638ec"
format_stamp: "Formatted at 2026-09-28 06:43:54 on dist-test-slave-b28r"
server_key: "3217f7e8a78955d89b0cfc10824f5a78"
server_key_iv: "2fca1e186bdacd3bca4e53977ae53480"
server_key_version: "encryptionkey@0"
I20260928 06:43:54.736707 6802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:43:54.755532 6802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:43:54.755810 6802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:43:54.755918 6802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:43:54.756127 6802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 06:43:54.756394 6802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 06:43:54.756423 6802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:54.756443 6802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 06:43:54.756459 6802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:54.761766 6802 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.67:44071
I20260928 06:43:54.761871 6929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.67:44071 every 8 connection(s)
I20260928 06:43:54.762121 6802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-2/data/info.pb
I20260928 06:43:54.766245 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 6802
I20260928 06:43:54.766376 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-2/wal/instance
I20260928 06:43:54.766580 19277 external_mini_cluster.cc:1428] Setting key 183dddc28da37ff2b126d63aa8657052
I20260928 06:43:54.767343 6520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:54.762869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.67:34573 (local address 127.18.211.126:33203)
0928 06:43:54.762926 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:54.762928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:54.763175 (+ 247us) server_negotiation.cc:407] Connection header received
0928 06:43:54.763651 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:54.763652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:54.763664 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:54.763721 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:54.764388 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.764903 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.765843 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.766068 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.766121 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:54.767055 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:54.767056 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:54.767057 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:43:54.767061 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:54.767077 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:43:54.767102 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:54.767149 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:54.767229 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 06:43:54.767264 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 06:43:54.767580 6932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:54.762860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:33203 (local address 127.18.211.67:34573)
0928 06:43:54.763031 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:54.763065 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:54.763570 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:54.763776 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:54.763779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:54.763793 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:43:54.764289 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:54.764295 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.764978 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.764978 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:54.765757 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:54.765759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.766142 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.766144 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:54.766301 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:54.766915 (+ 614us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 06:43:54.766915 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:43:54.766944 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 06:43:54.766967 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:43:54.767150 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:43:54.767156 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:43:54.767159 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 06:43:54.767228 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 06:43:54.767379 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":78,"threads_started":1}
I20260928 06:43:54.768010 6930 heartbeater.cc:347] Connected to a master server at 127.18.211.126:33203
I20260928 06:43:54.768086 6930 heartbeater.cc:464] Registering TS with master...
I20260928 06:43:54.768239 6930 heartbeater.cc:511] Master 127.18.211.126:33203 requested a full tablet report, sending...
I20260928 06:43:54.768597 6480 ts_manager.cc:194] Registered new tserver with Master: df5a42072ddb47c7bf556949ea5638ec (127.18.211.67:44071)
I20260928 06:43:54.768975 6480 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.211.67:34573
I20260928 06:43:54.778228 19277 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 06:43:54.782764 6520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:54.779556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53378 (local address 127.18.211.126:33203)
0928 06:43:54.779614 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:54.779616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:54.779676 (+ 60us) server_negotiation.cc:407] Connection header received
0928 06:43:54.779732 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:54.779733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:54.779746 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:54.779807 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:54.779948 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.780733 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.781506 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.781805 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.781881 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:54.782538 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:54.782539 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:54.782539 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:43:54.782544 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:54.782557 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:43:54.782562 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:54.782615 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:54.782655 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 06:43:54.782681 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:43:54.784456 6480 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 06:43:54.786649 6480 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 06:43:54.794116 6939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:54.788901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:41679 (local address 127.0.0.1:58500)
0928 06:43:54.789531 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:54.789564 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:54.789623 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:54.789927 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:54.789928 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:54.789930 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:43:54.789981 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:54.789983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.791323 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.791324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:54.792715 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:54.792718 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.793785 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.793786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:54.793874 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:54.793891 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 06:43:54.793963 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 06:43:54.794034 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"mutex_wait_us":59,"thread_start_us":101,"threads_started":1}
I20260928 06:43:54.794159 6943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:54.789280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51736 (local address 127.18.211.67:44071)
0928 06:43:54.789467 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:54.789471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:54.789939 (+ 468us) server_negotiation.cc:407] Connection header received
0928 06:43:54.789986 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:54.789988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:54.790004 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:54.790075 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:43:54.790292 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.791566 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.792824 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.793563 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.793645 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:54.793742 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:54.794027 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 06:43:54.794067 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":87,"threads_started":1}
I20260928 06:43:54.794469 6938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:54.788890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:36695 (local address 127.0.0.1:49174)
0928 06:43:54.789240 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:54.789294 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:54.789356 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:54.789897 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:54.789901 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:54.789903 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:43:54.789973 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:54.789975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.791343 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.791344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:54.792876 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:54.792880 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.794216 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.794217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:54.794298 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:54.794302 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 06:43:54.794351 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 06:43:54.794391 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":35,"thread_start_us":81,"threads_started":1}
I20260928 06:43:54.794696 6941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:54.789109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:44071 (local address 127.0.0.1:51736)
0928 06:43:54.789807 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:54.789817 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:54.789872 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:54.790158 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:54.790159 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:54.790161 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:43:54.790194 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:54.790195 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.791661 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.791662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:54.792733 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:54.792735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.793783 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.793784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:54.793874 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:54.793891 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 06:43:54.793963 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 06:43:54.794464 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":442,"thread_start_us":72,"threads_started":1}
I20260928 06:43:54.794792 6940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:54.789009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49174 (local address 127.18.211.66:36695)
0928 06:43:54.789530 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:54.789533 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:54.789545 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:43:54.789600 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:54.789602 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:54.789620 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:54.789703 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:43:54.790059 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.791243 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.792974 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.794035 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.794090 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:54.794184 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:54.794665 (+ 481us) server_negotiation.cc:299] Negotiation successful
0928 06:43:54.794712 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":100,"threads_started":1}
I20260928 06:43:54.795184 6942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:54.789192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58500 (local address 127.18.211.65:41679)
0928 06:43:54.789428 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:54.789433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:54.789704 (+ 271us) server_negotiation.cc:407] Connection header received
0928 06:43:54.789757 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:54.789760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:54.789776 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:54.789851 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:43:54.790067 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.791221 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.792827 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.793557 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.793645 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:54.793742 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:54.794717 (+ 975us) server_negotiation.cc:299] Negotiation successful
0928 06:43:54.795075 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":102,"threads_started":1}
I20260928 06:43:54.795475 6733 tablet_service.cc:1467] Processing CreateTablet for tablet 800b0bb5d3274638bb8ee95ec0157078 (DEFAULT_TABLE table=test-table [id=5cc6aa169efd490c92b4e314a635915d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:43:54.795470 6864 tablet_service.cc:1467] Processing CreateTablet for tablet 800b0bb5d3274638bb8ee95ec0157078 (DEFAULT_TABLE table=test-table [id=5cc6aa169efd490c92b4e314a635915d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:43:54.795838 6733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 800b0bb5d3274638bb8ee95ec0157078. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:43:54.795845 6864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 800b0bb5d3274638bb8ee95ec0157078. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:43:54.795903 6602 tablet_service.cc:1467] Processing CreateTablet for tablet 800b0bb5d3274638bb8ee95ec0157078 (DEFAULT_TABLE table=test-table [id=5cc6aa169efd490c92b4e314a635915d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 06:43:54.796204 6602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 800b0bb5d3274638bb8ee95ec0157078. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:43:54.799532 6946 tablet_bootstrap.cc:492] T 800b0bb5d3274638bb8ee95ec0157078 P 1d216ab3251f49259e63430e1bd6a387: Bootstrap starting.
I20260928 06:43:54.799909 6944 tablet_bootstrap.cc:492] T 800b0bb5d3274638bb8ee95ec0157078 P 8f2c5a26fa0b40258c15c4fff9cfa886: Bootstrap starting.
I20260928 06:43:54.800184 6946 tablet_bootstrap.cc:654] T 800b0bb5d3274638bb8ee95ec0157078 P 1d216ab3251f49259e63430e1bd6a387: Neither blocks nor log segments found. Creating new log.
I20260928 06:43:54.800475 6946 log.cc:824] T 800b0bb5d3274638bb8ee95ec0157078 P 1d216ab3251f49259e63430e1bd6a387: Log is configured to *not* fsync() on all Append() calls
I20260928 06:43:54.800562 6944 tablet_bootstrap.cc:654] T 800b0bb5d3274638bb8ee95ec0157078 P 8f2c5a26fa0b40258c15c4fff9cfa886: Neither blocks nor log segments found. Creating new log.
I20260928 06:43:54.800827 6945 tablet_bootstrap.cc:492] T 800b0bb5d3274638bb8ee95ec0157078 P df5a42072ddb47c7bf556949ea5638ec: Bootstrap starting.
I20260928 06:43:54.800859 6944 log.cc:824] T 800b0bb5d3274638bb8ee95ec0157078 P 8f2c5a26fa0b40258c15c4fff9cfa886: Log is configured to *not* fsync() on all Append() calls
I20260928 06:43:54.801344 6946 tablet_bootstrap.cc:492] T 800b0bb5d3274638bb8ee95ec0157078 P 1d216ab3251f49259e63430e1bd6a387: No bootstrap required, opened a new log
I20260928 06:43:54.801467 6946 ts_tablet_manager.cc:1397] T 800b0bb5d3274638bb8ee95ec0157078 P 1d216ab3251f49259e63430e1bd6a387: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 06:43:54.801486 6945 tablet_bootstrap.cc:654] T 800b0bb5d3274638bb8ee95ec0157078 P df5a42072ddb47c7bf556949ea5638ec: Neither blocks nor log segments found. Creating new log.
I20260928 06:43:54.801559 6944 tablet_bootstrap.cc:492] T 800b0bb5d3274638bb8ee95ec0157078 P 8f2c5a26fa0b40258c15c4fff9cfa886: No bootstrap required, opened a new log
I20260928 06:43:54.801627 6944 ts_tablet_manager.cc:1397] T 800b0bb5d3274638bb8ee95ec0157078 P 8f2c5a26fa0b40258c15c4fff9cfa886: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:43:54.801811 6945 log.cc:824] T 800b0bb5d3274638bb8ee95ec0157078 P df5a42072ddb47c7bf556949ea5638ec: Log is configured to *not* fsync() on all Append() calls
I20260928 06:43:54.802542 6945 tablet_bootstrap.cc:492] T 800b0bb5d3274638bb8ee95ec0157078 P df5a42072ddb47c7bf556949ea5638ec: No bootstrap required, opened a new log
I20260928 06:43:54.802628 6945 ts_tablet_manager.cc:1397] T 800b0bb5d3274638bb8ee95ec0157078 P df5a42072ddb47c7bf556949ea5638ec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 06:43:54.804244 6944 raft_consensus.cc:348] T 800b0bb5d3274638bb8ee95ec0157078 P 8f2c5a26fa0b40258c15c4fff9cfa886 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2c5a26fa0b40258c15c4fff9cfa886" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36695 } } peers { permanent_uuid: "1d216ab3251f49259e63430e1bd6a387" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41679 } } peers { permanent_uuid: "df5a42072ddb47c7bf556949ea5638ec" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 44071 } }
I20260928 06:43:54.804437 6944 raft_consensus.cc:374] T 800b0bb5d3274638bb8ee95ec0157078 P 8f2c5a26fa0b40258c15c4fff9cfa886 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:43:54.804469 6944 raft_consensus.cc:729] T 800b0bb5d3274638bb8ee95ec0157078 P 8f2c5a26fa0b40258c15c4fff9cfa886 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2c5a26fa0b40258c15c4fff9cfa886, State: Initialized, Role: FOLLOWER
I20260928 06:43:54.804580 6944 consensus_queue.cc:260] T 800b0bb5d3274638bb8ee95ec0157078 P 8f2c5a26fa0b40258c15c4fff9cfa886 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2c5a26fa0b40258c15c4fff9cfa886" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36695 } } peers { permanent_uuid: "1d216ab3251f49259e63430e1bd6a387" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41679 } } peers { permanent_uuid: "df5a42072ddb47c7bf556949ea5638ec" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 44071 } }
I20260928 06:43:54.804849 6944 ts_tablet_manager.cc:1428] T 800b0bb5d3274638bb8ee95ec0157078 P 8f2c5a26fa0b40258c15c4fff9cfa886: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 06:43:54.804656 6946 raft_consensus.cc:348] T 800b0bb5d3274638bb8ee95ec0157078 P 1d216ab3251f49259e63430e1bd6a387 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2c5a26fa0b40258c15c4fff9cfa886" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36695 } } peers { permanent_uuid: "1d216ab3251f49259e63430e1bd6a387" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41679 } } peers { permanent_uuid: "df5a42072ddb47c7bf556949ea5638ec" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 44071 } }
I20260928 06:43:54.805047 6946 raft_consensus.cc:374] T 800b0bb5d3274638bb8ee95ec0157078 P 1d216ab3251f49259e63430e1bd6a387 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:43:54.805135 6946 raft_consensus.cc:729] T 800b0bb5d3274638bb8ee95ec0157078 P 1d216ab3251f49259e63430e1bd6a387 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d216ab3251f49259e63430e1bd6a387, State: Initialized, Role: FOLLOWER
I20260928 06:43:54.805333 6799 heartbeater.cc:503] Master 127.18.211.126:33203 was elected leader, sending a full tablet report...
I20260928 06:43:54.805284 6945 raft_consensus.cc:348] T 800b0bb5d3274638bb8ee95ec0157078 P df5a42072ddb47c7bf556949ea5638ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2c5a26fa0b40258c15c4fff9cfa886" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36695 } } peers { permanent_uuid: "1d216ab3251f49259e63430e1bd6a387" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41679 } } peers { permanent_uuid: "df5a42072ddb47c7bf556949ea5638ec" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 44071 } }
I20260928 06:43:54.805437 6945 raft_consensus.cc:374] T 800b0bb5d3274638bb8ee95ec0157078 P df5a42072ddb47c7bf556949ea5638ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:43:54.805486 6945 raft_consensus.cc:729] T 800b0bb5d3274638bb8ee95ec0157078 P df5a42072ddb47c7bf556949ea5638ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df5a42072ddb47c7bf556949ea5638ec, State: Initialized, Role: FOLLOWER
I20260928 06:43:54.805335 6946 consensus_queue.cc:260] T 800b0bb5d3274638bb8ee95ec0157078 P 1d216ab3251f49259e63430e1bd6a387 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2c5a26fa0b40258c15c4fff9cfa886" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36695 } } peers { permanent_uuid: "1d216ab3251f49259e63430e1bd6a387" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41679 } } peers { permanent_uuid: "df5a42072ddb47c7bf556949ea5638ec" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 44071 } }
I20260928 06:43:54.805655 6945 consensus_queue.cc:260] T 800b0bb5d3274638bb8ee95ec0157078 P df5a42072ddb47c7bf556949ea5638ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2c5a26fa0b40258c15c4fff9cfa886" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36695 } } peers { permanent_uuid: "1d216ab3251f49259e63430e1bd6a387" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41679 } } peers { permanent_uuid: "df5a42072ddb47c7bf556949ea5638ec" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 44071 } }
I20260928 06:43:54.805899 6946 ts_tablet_manager.cc:1428] T 800b0bb5d3274638bb8ee95ec0157078 P 1d216ab3251f49259e63430e1bd6a387: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260928 06:43:54.805898 6945 ts_tablet_manager.cc:1428] T 800b0bb5d3274638bb8ee95ec0157078 P df5a42072ddb47c7bf556949ea5638ec: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260928 06:43:54.806039 6930 heartbeater.cc:503] Master 127.18.211.126:33203 was elected leader, sending a full tablet report...
I20260928 06:43:54.806056 6668 heartbeater.cc:503] Master 127.18.211.126:33203 was elected leader, sending a full tablet report...
W20260928 06:43:54.884078 6800 tablet.cc:2369] T 800b0bb5d3274638bb8ee95ec0157078 P 8f2c5a26fa0b40258c15c4fff9cfa886: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 06:43:54.935840 6951 raft_consensus.cc:482] T 800b0bb5d3274638bb8ee95ec0157078 P 1d216ab3251f49259e63430e1bd6a387 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 06:43:54.935985 6951 raft_consensus.cc:504] T 800b0bb5d3274638bb8ee95ec0157078 P 1d216ab3251f49259e63430e1bd6a387 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2c5a26fa0b40258c15c4fff9cfa886" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36695 } } peers { permanent_uuid: "1d216ab3251f49259e63430e1bd6a387" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41679 } } peers { permanent_uuid: "df5a42072ddb47c7bf556949ea5638ec" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 44071 } }
I20260928 06:43:54.936358 6951 leader_election.cc:288] T 800b0bb5d3274638bb8ee95ec0157078 P 1d216ab3251f49259e63430e1bd6a387 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2c5a26fa0b40258c15c4fff9cfa886 (127.18.211.66:36695), df5a42072ddb47c7bf556949ea5638ec (127.18.211.67:44071)
I20260928 06:43:54.940025 6940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:54.936637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:37185 (local address 127.18.211.66:36695)
0928 06:43:54.936688 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:54.936690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:54.936700 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:43:54.936732 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:54.936733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:54.936743 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:54.936823 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:43:54.936979 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.937585 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.939099 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.939670 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.939737 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:54.939744 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:54.939898 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 06:43:54.939953 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:43:54.940025 6943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:54.936745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.211.65:39409 (local address 127.18.211.67:44071)
0928 06:43:54.936796 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:54.936798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:54.936896 (+ 98us) server_negotiation.cc:407] Connection header received
0928 06:43:54.936944 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:54.936945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:54.936956 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:54.937000 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 06:43:54.937140 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.937736 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.939103 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.939663 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.939737 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:54.939744 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:54.939897 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 06:43:54.939953 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 06:43:54.940032 6670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:54.936547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:36695 (local address 127.18.211.65:37185)
0928 06:43:54.936594 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:54.936606 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:54.936653 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:54.936829 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:54.936830 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:54.936832 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:43:54.936903 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:54.936905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.937674 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.937675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:54.938978 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:54.938982 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.939740 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.939741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:54.939823 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:54.939830 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 06:43:54.939888 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 06:43:54.939943 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":16}
I20260928 06:43:54.940037 6953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:54.936648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.67:44071 (local address 127.18.211.65:39409)
0928 06:43:54.936824 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:54.936836 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:43:54.936896 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:43:54.937013 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:43:54.937014 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:43:54.937015 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 06:43:54.937083 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:54.937084 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.937793 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.937794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:54.938981 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:43:54.938982 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.939740 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.939741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:43:54.939823 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:54.939830 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 06:43:54.939888 (+ 58us) client_negotiation.cc:240] Negotiation successful
0928 06:43:54.939940 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":29,"thread_start_us":102,"threads_started":1}
I20260928 06:43:54.940443 6884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "800b0bb5d3274638bb8ee95ec0157078" candidate_uuid: "1d216ab3251f49259e63430e1bd6a387" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df5a42072ddb47c7bf556949ea5638ec" is_pre_election: true
I20260928 06:43:54.940490 6753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "800b0bb5d3274638bb8ee95ec0157078" candidate_uuid: "1d216ab3251f49259e63430e1bd6a387" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2c5a26fa0b40258c15c4fff9cfa886" is_pre_election: true
I20260928 06:43:54.940585 6884 raft_consensus.cc:2454] T 800b0bb5d3274638bb8ee95ec0157078 P df5a42072ddb47c7bf556949ea5638ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d216ab3251f49259e63430e1bd6a387 in term 0.
I20260928 06:43:54.940637 6753 raft_consensus.cc:2454] T 800b0bb5d3274638bb8ee95ec0157078 P 8f2c5a26fa0b40258c15c4fff9cfa886 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d216ab3251f49259e63430e1bd6a387 in term 0.
I20260928 06:43:54.940834 6557 leader_election.cc:302] T 800b0bb5d3274638bb8ee95ec0157078 P 1d216ab3251f49259e63430e1bd6a387 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d216ab3251f49259e63430e1bd6a387, df5a42072ddb47c7bf556949ea5638ec; no voters:
I20260928 06:43:54.940963 6951 raft_consensus.cc:2781] T 800b0bb5d3274638bb8ee95ec0157078 P 1d216ab3251f49259e63430e1bd6a387 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 06:43:54.941030 6951 raft_consensus.cc:482] T 800b0bb5d3274638bb8ee95ec0157078 P 1d216ab3251f49259e63430e1bd6a387 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 06:43:54.941053 6951 raft_consensus.cc:3037] T 800b0bb5d3274638bb8ee95ec0157078 P 1d216ab3251f49259e63430e1bd6a387 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:43:54.941756 6951 raft_consensus.cc:504] T 800b0bb5d3274638bb8ee95ec0157078 P 1d216ab3251f49259e63430e1bd6a387 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2c5a26fa0b40258c15c4fff9cfa886" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36695 } } peers { permanent_uuid: "1d216ab3251f49259e63430e1bd6a387" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41679 } } peers { permanent_uuid: "df5a42072ddb47c7bf556949ea5638ec" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 44071 } }
I20260928 06:43:54.941874 6951 leader_election.cc:288] T 800b0bb5d3274638bb8ee95ec0157078 P 1d216ab3251f49259e63430e1bd6a387 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2c5a26fa0b40258c15c4fff9cfa886 (127.18.211.66:36695), df5a42072ddb47c7bf556949ea5638ec (127.18.211.67:44071)
I20260928 06:43:54.942058 6753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "800b0bb5d3274638bb8ee95ec0157078" candidate_uuid: "1d216ab3251f49259e63430e1bd6a387" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2c5a26fa0b40258c15c4fff9cfa886"
I20260928 06:43:54.942085 6884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "800b0bb5d3274638bb8ee95ec0157078" candidate_uuid: "1d216ab3251f49259e63430e1bd6a387" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df5a42072ddb47c7bf556949ea5638ec"
I20260928 06:43:54.942145 6884 raft_consensus.cc:3037] T 800b0bb5d3274638bb8ee95ec0157078 P df5a42072ddb47c7bf556949ea5638ec [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:43:54.942150 6753 raft_consensus.cc:3037] T 800b0bb5d3274638bb8ee95ec0157078 P 8f2c5a26fa0b40258c15c4fff9cfa886 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:43:54.942854 6884 raft_consensus.cc:2454] T 800b0bb5d3274638bb8ee95ec0157078 P df5a42072ddb47c7bf556949ea5638ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d216ab3251f49259e63430e1bd6a387 in term 1.
I20260928 06:43:54.942926 6753 raft_consensus.cc:2454] T 800b0bb5d3274638bb8ee95ec0157078 P 8f2c5a26fa0b40258c15c4fff9cfa886 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d216ab3251f49259e63430e1bd6a387 in term 1.
I20260928 06:43:54.943105 6557 leader_election.cc:302] T 800b0bb5d3274638bb8ee95ec0157078 P 1d216ab3251f49259e63430e1bd6a387 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d216ab3251f49259e63430e1bd6a387, df5a42072ddb47c7bf556949ea5638ec; no voters:
I20260928 06:43:54.943213 6951 raft_consensus.cc:2781] T 800b0bb5d3274638bb8ee95ec0157078 P 1d216ab3251f49259e63430e1bd6a387 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:43:54.943385 6951 raft_consensus.cc:686] T 800b0bb5d3274638bb8ee95ec0157078 P 1d216ab3251f49259e63430e1bd6a387 [term 1 LEADER]: Becoming Leader. State: Replica: 1d216ab3251f49259e63430e1bd6a387, State: Running, Role: LEADER
I20260928 06:43:54.943467 6951 consensus_queue.cc:237] T 800b0bb5d3274638bb8ee95ec0157078 P 1d216ab3251f49259e63430e1bd6a387 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2c5a26fa0b40258c15c4fff9cfa886" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36695 } } peers { permanent_uuid: "1d216ab3251f49259e63430e1bd6a387" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41679 } } peers { permanent_uuid: "df5a42072ddb47c7bf556949ea5638ec" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 44071 } }
I20260928 06:43:54.944330 6478 catalog_manager.cc:5495] T 800b0bb5d3274638bb8ee95ec0157078 P 1d216ab3251f49259e63430e1bd6a387 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d216ab3251f49259e63430e1bd6a387 (127.18.211.65). New cstate: current_term: 1 leader_uuid: "1d216ab3251f49259e63430e1bd6a387" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2c5a26fa0b40258c15c4fff9cfa886" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d216ab3251f49259e63430e1bd6a387" member_type: VOTER last_known_addr { host: "127.18.211.65" port: 41679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df5a42072ddb47c7bf556949ea5638ec" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 44071 } health_report { overall_health: UNKNOWN } } }
I20260928 06:43:54.977849 6942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:54.975568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58502 (local address 127.18.211.65:41679)
0928 06:43:54.975632 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:54.975634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:54.975642 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:43:54.975683 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:54.975684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:54.975705 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:54.975764 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:43:54.975904 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.976396 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.977123 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:54.977335 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:54.977390 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:54.977540 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:43:54.977648 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:43:54.977720 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:54.977741 (+ 21us) server_negotiation.cc:299] Negotiation successful
0928 06:43:54.977761 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 06:43:54.979696 6884 raft_consensus.cc:1261] T 800b0bb5d3274638bb8ee95ec0157078 P df5a42072ddb47c7bf556949ea5638ec [term 1 FOLLOWER]: Refusing update from remote peer 1d216ab3251f49259e63430e1bd6a387: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:43:54.979717 6753 raft_consensus.cc:1261] T 800b0bb5d3274638bb8ee95ec0157078 P 8f2c5a26fa0b40258c15c4fff9cfa886 [term 1 FOLLOWER]: Refusing update from remote peer 1d216ab3251f49259e63430e1bd6a387: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:43:54.980015 6951 consensus_queue.cc:1035] T 800b0bb5d3274638bb8ee95ec0157078 P 1d216ab3251f49259e63430e1bd6a387 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df5a42072ddb47c7bf556949ea5638ec" member_type: VOTER last_known_addr { host: "127.18.211.67" port: 44071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:43:54.980182 6954 consensus_queue.cc:1035] T 800b0bb5d3274638bb8ee95ec0157078 P 1d216ab3251f49259e63430e1bd6a387 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f2c5a26fa0b40258c15c4fff9cfa886" member_type: VOTER last_known_addr { host: "127.18.211.66" port: 36695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:43:54.982502 6957 mvcc.cc:205] Tried to move back new op lower bound from 7334206812073955328 to 7334206811928944640. Current Snapshot: MvccSnapshot[applied={T|T < 7334206812073955328}]
I20260928 06:43:54.982692 6958 mvcc.cc:205] Tried to move back new op lower bound from 7334206812073955328 to 7334206811928944640. Current Snapshot: MvccSnapshot[applied={T|T < 7334206812073955328}]
I20260928 06:43:54.983002 6959 mvcc.cc:205] Tried to move back new op lower bound from 7334206812073955328 to 7334206811928944640. Current Snapshot: MvccSnapshot[applied={T|T < 7334206812073955328}]
I20260928 06:43:54.984689 6479 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:43:54.984802 6479 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 06:43:54.985651 6479 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 878380558fea47fbaf0630c4a4f5870d: Sending DeleteTablet for 3 replicas of tablet 800b0bb5d3274638bb8ee95ec0157078
I20260928 06:43:54.985920 6733 tablet_service.cc:1514] Processing DeleteTablet for tablet 800b0bb5d3274638bb8ee95ec0157078 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:43:54 UTC) from {username='slave'} at 127.0.0.1:49174
I20260928 06:43:54.985963 6602 tablet_service.cc:1514] Processing DeleteTablet for tablet 800b0bb5d3274638bb8ee95ec0157078 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:43:54 UTC) from {username='slave'} at 127.0.0.1:58500
I20260928 06:43:54.986140 6864 tablet_service.cc:1514] Processing DeleteTablet for tablet 800b0bb5d3274638bb8ee95ec0157078 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:43:54 UTC) from {username='slave'} at 127.0.0.1:51736
I20260928 06:43:54.986361 6601 tablet_service.cc:1514] Processing DeleteTablet for tablet 800b0bb5d3274638bb8ee95ec0157078 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 06:43:54 UTC) from {username='slave'} at 127.0.0.1:58500
I20260928 06:43:54.986406 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 6540
I20260928 06:43:54.986375 6968 tablet_replica.cc:330] T 800b0bb5d3274638bb8ee95ec0157078 P df5a42072ddb47c7bf556949ea5638ec: stopping tablet replica
I20260928 06:43:54.986485 6966 tablet_replica.cc:330] T 800b0bb5d3274638bb8ee95ec0157078 P 8f2c5a26fa0b40258c15c4fff9cfa886: stopping tablet replica
I20260928 06:43:54.986603 6968 raft_consensus.cc:2229] T 800b0bb5d3274638bb8ee95ec0157078 P df5a42072ddb47c7bf556949ea5638ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:43:54.986738 6968 raft_consensus.cc:2258] T 800b0bb5d3274638bb8ee95ec0157078 P df5a42072ddb47c7bf556949ea5638ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:43:54.988404 6966 raft_consensus.cc:2229] T 800b0bb5d3274638bb8ee95ec0157078 P 8f2c5a26fa0b40258c15c4fff9cfa886 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 06:43:54.988529 6966 raft_consensus.cc:2258] T 800b0bb5d3274638bb8ee95ec0157078 P 8f2c5a26fa0b40258c15c4fff9cfa886 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 06:43:54.988806 6968 ts_tablet_manager.cc:1905] T 800b0bb5d3274638bb8ee95ec0157078 P df5a42072ddb47c7bf556949ea5638ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 06:43:54.990206 6968 ts_tablet_manager.cc:1918] T 800b0bb5d3274638bb8ee95ec0157078 P df5a42072ddb47c7bf556949ea5638ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 06:43:54.990269 6968 log.cc:1196] T 800b0bb5d3274638bb8ee95ec0157078 P df5a42072ddb47c7bf556949ea5638ec: Deleting WAL directory at /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790577602522428-19277-0/minicluster-data/ts-2/wal/wals/800b0bb5d3274638bb8ee95ec0157078
I20260928 06:43:54.990479 6968 ts_tablet_manager.cc:1939] T 800b0bb5d3274638bb8ee95ec0157078 P df5a42072ddb47c7bf556949ea5638ec: Deleting consensus metadata
I20260928 06:43:54.990820 6966 ts_tablet_manager.cc:1905] T 800b0bb5d3274638bb8ee95ec0157078 P 8f2c5a26fa0b40258c15c4fff9cfa886: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 06:43:54.991200 6467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.211.65:41679 (error 108)
I20260928 06:43:54.991465 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 6671
I20260928 06:43:54.991663 6938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:54.991530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.65:41679 (local address 127.0.0.1:58516)
0928 06:43:54.991584 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:54.991622 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.65:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260928 06:43:54.991770 6467 catalog_manager.cc:4584] TS 1d216ab3251f49259e63430e1bd6a387 (127.18.211.65:41679): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 800b0bb5d3274638bb8ee95ec0157078: Network error: Client connection negotiation failed: client connection to 127.18.211.65:41679: connect: Connection refused (error 111)
I20260928 06:43:54.992290 6466 catalog_manager.cc:4841] TS df5a42072ddb47c7bf556949ea5638ec (127.18.211.67:44071): tablet 800b0bb5d3274638bb8ee95ec0157078 (table test-table [id=5cc6aa169efd490c92b4e314a635915d]) successfully deleted
I20260928 06:43:54.996766 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 6802
I20260928 06:43:54.996815 6938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:54.996677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.66:36695 (local address 127.0.0.1:49176)
0928 06:43:54.996743 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0928 06:43:54.996781 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.66:36695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 06:43:55.001149 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 6449
2026-09-28T06:43:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (794 ms)
[----------] 6 tests from EncryptionPolicyTest (8123 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:43:55 dist-test-slave-b28r krb5kdc[6972](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:43:55 dist-test-slave-b28r krb5kdc[6972](info): set up 2 sockets
Sep 28 06:43:55 dist-test-slave-b28r krb5kdc[6972](info): commencing operation
krb5kdc: starting...
W20260928 06:43:55.951881 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:43:56 dist-test-slave-b28r krb5kdc[6972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:43:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:43:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:43:56.062448 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34683
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:43:56.141819 6985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:43:56.142040 6985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:43:56.142071 6985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:43:56.143277 6985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:43:56.143323 6985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:43:56.143342 6985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:43:56.143358 6985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:43:56.143373 6985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:43:56.144856 6985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34683
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:43:56.145102 6985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:43:56.145311 6985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:43:56.150059 6993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:56.150151 6985 server_base.cc:1034] running on GCE node
W20260928 06:43:56.150072 6991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:56.150072 6990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:56.150568 6985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:43:56.150852 6985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:43:56.152021 6985 hybrid_clock.cc:648] HybridClock initialized: now 1790577836152005 us; error 26 us; skew 500 ppm
Sep 28 06:43:56 dist-test-slave-b28r krb5kdc[6972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577836, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:43:56.154489 6985 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:43:56.154883 6985 webserver.cc:460] Webserver started at http://127.0.0.1:38971/ using document root <none> and password file <none>
I20260928 06:43:56.155067 6985 fs_manager.cc:359] Metadata directory not provided
I20260928 06:43:56.155144 6985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:43:56.155285 6985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:43:56.156293 6985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "8e1dfd86d3384675b50f6158833b110c"
format_stamp: "Formatted at 2026-09-28 06:43:56 on dist-test-slave-b28r"
server_key: "cacd347937f004780382ea4557e045fc"
server_key_iv: "52c55ccfa47414f617347c4b4ee95ea9"
server_key_version: "encryptionkey@0"
I20260928 06:43:56.156585 6985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "8e1dfd86d3384675b50f6158833b110c"
format_stamp: "Formatted at 2026-09-28 06:43:56 on dist-test-slave-b28r"
server_key: "cacd347937f004780382ea4557e045fc"
server_key_iv: "52c55ccfa47414f617347c4b4ee95ea9"
server_key_version: "encryptionkey@0"
I20260928 06:43:56.157805 6985 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:43:56.158555 7000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:56.158725 6985 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:43:56.158818 6985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "8e1dfd86d3384675b50f6158833b110c"
format_stamp: "Formatted at 2026-09-28 06:43:56 on dist-test-slave-b28r"
server_key: "cacd347937f004780382ea4557e045fc"
server_key_iv: "52c55ccfa47414f617347c4b4ee95ea9"
server_key_version: "encryptionkey@0"
I20260928 06:43:56.158874 6985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:43:56.178357 6985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:43:56.180557 6985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:43:56.180689 6985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:43:56.184770 6985 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34683
I20260928 06:43:56.184815 7052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34683 every 8 connection(s)
I20260928 06:43:56.185148 6985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:43:56.186049 7053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:43:56.186957 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 6985
I20260928 06:43:56.187109 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:43:56.187325 19277 external_mini_cluster.cc:1428] Setting key e0e71e531dda2e5229a8c06f7dca6fd6
I20260928 06:43:56.188725 7053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e1dfd86d3384675b50f6158833b110c: Bootstrap starting.
I20260928 06:43:56.189437 7053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e1dfd86d3384675b50f6158833b110c: Neither blocks nor log segments found. Creating new log.
I20260928 06:43:56.189738 7053 log.cc:824] T 00000000000000000000000000000000 P 8e1dfd86d3384675b50f6158833b110c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:43:56.190372 7053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e1dfd86d3384675b50f6158833b110c: No bootstrap required, opened a new log
I20260928 06:43:56.192659 7053 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e1dfd86d3384675b50f6158833b110c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e1dfd86d3384675b50f6158833b110c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34683 } }
I20260928 06:43:56.192819 7053 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e1dfd86d3384675b50f6158833b110c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:43:56.192884 7053 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e1dfd86d3384675b50f6158833b110c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e1dfd86d3384675b50f6158833b110c, State: Initialized, Role: FOLLOWER
I20260928 06:43:56.193005 7053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e1dfd86d3384675b50f6158833b110c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e1dfd86d3384675b50f6158833b110c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34683 } }
I20260928 06:43:56.193065 7053 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e1dfd86d3384675b50f6158833b110c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:43:56.193094 7053 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e1dfd86d3384675b50f6158833b110c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:43:56.193138 7053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e1dfd86d3384675b50f6158833b110c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:43:56.193696 7053 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e1dfd86d3384675b50f6158833b110c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e1dfd86d3384675b50f6158833b110c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34683 } }
I20260928 06:43:56.193801 7053 leader_election.cc:302] T 00000000000000000000000000000000 P 8e1dfd86d3384675b50f6158833b110c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e1dfd86d3384675b50f6158833b110c; no voters:
I20260928 06:43:56.193967 7053 leader_election.cc:288] T 00000000000000000000000000000000 P 8e1dfd86d3384675b50f6158833b110c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:43:56.194005 7058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e1dfd86d3384675b50f6158833b110c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:43:56.194146 7058 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e1dfd86d3384675b50f6158833b110c [term 1 LEADER]: Becoming Leader. State: Replica: 8e1dfd86d3384675b50f6158833b110c, State: Running, Role: LEADER
I20260928 06:43:56.194219 7053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e1dfd86d3384675b50f6158833b110c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:43:56.194273 7058 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e1dfd86d3384675b50f6158833b110c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e1dfd86d3384675b50f6158833b110c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34683 } }
Sep 28 06:43:56 dist-test-slave-b28r krb5kdc[6972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 06:43:56.194624 7058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e1dfd86d3384675b50f6158833b110c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e1dfd86d3384675b50f6158833b110c. Latest consensus state: current_term: 1 leader_uuid: "8e1dfd86d3384675b50f6158833b110c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e1dfd86d3384675b50f6158833b110c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34683 } } }
I20260928 06:43:56.194717 7058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e1dfd86d3384675b50f6158833b110c [sys.catalog]: This master's current role is: LEADER
I20260928 06:43:56.194633 7060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e1dfd86d3384675b50f6158833b110c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e1dfd86d3384675b50f6158833b110c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e1dfd86d3384675b50f6158833b110c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34683 } } }
I20260928 06:43:56.194809 7060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e1dfd86d3384675b50f6158833b110c [sys.catalog]: This master's current role is: LEADER
I20260928 06:43:56.195446 7065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:43:56.195899 7065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:43:56.197690 7065 catalog_manager.cc:1347] Generated new cluster ID: e87d1a494890468b80d4aa12d4694647
I20260928 06:43:56.197737 7065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:43:56.200052 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:56.187992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41967 (local address 127.0.0.1:34683)
0928 06:43:56.188185 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:56.188194 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:56.188219 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:43:56.188664 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:56.188671 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:56.188718 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:56.188861 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:56.189634 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:56.190638 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:56.191752 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:56.192158 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:56.192287 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:56.194936 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:56.194940 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:56.194941 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:43:56.194954 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:56.196885 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:56.197350 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:56.197353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:56.197353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:56.197446 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:56.199156 (+ 1710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:56.199157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:56.199157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:56.199450 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:56.199599 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:56.199748 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 06:43:56.199851 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":92,"threads_started":1}
I20260928 06:43:56.200956 19277 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:43:56.209056 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:56.202272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57811 (local address 127.0.0.1:34683)
0928 06:43:56.202307 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:56.202309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:56.202413 (+ 104us) server_negotiation.cc:407] Connection header received
0928 06:43:56.202473 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:56.202474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:56.202482 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:56.202532 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:56.203144 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:56.203901 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:56.204863 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:56.205180 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:56.205233 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:56.206504 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:56.206505 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:56.206506 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:43:56.206512 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:56.207975 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:56.208367 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:56.208368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:56.208368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:56.208415 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:56.208605 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:56.208605 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:56.208606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:56.208710 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:56.208795 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:56.208917 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 06:43:56.208972 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:43:56.227969 7065 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:43:56.228538 7065 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:43:56.243505 7065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e1dfd86d3384675b50f6158833b110c: Generated new TSK 0
I20260928 06:43:56.243716 7065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:43:56.255389 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 6985
W20260928 06:43:56.259548 6970 connection.cc:535] client connection to 127.0.0.1:34683 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34683): Connection reset by peer (error 104)
2026-09-28T06:43:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1253 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:43:56 dist-test-slave-b28r krb5kdc[7084](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:43:56 dist-test-slave-b28r krb5kdc[7084](info): set up 2 sockets
Sep 28 06:43:56 dist-test-slave-b28r krb5kdc[7084](info): commencing operation
krb5kdc: starting...
W20260928 06:43:57.207939 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:43:57 dist-test-slave-b28r krb5kdc[7084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:43:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:43:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:43:57.319933 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38239
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:43:57.400252 7097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:43:57.400455 7097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:43:57.400475 7097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:43:57.401552 7097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:43:57.401584 7097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:43:57.401597 7097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:43:57.401607 7097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:43:57.401618 7097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:43:57.402984 7097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38239
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:43:57.403196 7097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:43:57.403379 7097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:43:57.407835 7103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:57.407815 7102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:57.408039 7105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:57.407951 7097 server_base.cc:1034] running on GCE node
I20260928 06:43:57.408277 7097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:43:57.408520 7097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:43:57.409663 7097 hybrid_clock.cc:648] HybridClock initialized: now 1790577837409647 us; error 33 us; skew 500 ppm
Sep 28 06:43:57 dist-test-slave-b28r krb5kdc[7084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577837, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:43:57.415625 7097 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:43:57.416038 7097 webserver.cc:460] Webserver started at http://127.0.0.1:45381/ using document root <none> and password file <none>
I20260928 06:43:57.416282 7097 fs_manager.cc:359] Metadata directory not provided
I20260928 06:43:57.416333 7097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:43:57.416455 7097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:43:57.417461 7097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "011f174d16864b9e8c1c76e59686e04a"
format_stamp: "Formatted at 2026-09-28 06:43:57 on dist-test-slave-b28r"
server_key: "045695a7373324a2af473eaaaec2e5de"
server_key_iv: "ba467501683e5efd2dcda13cf681fe0b"
server_key_version: "encryptionkey@0"
I20260928 06:43:57.417784 7097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "011f174d16864b9e8c1c76e59686e04a"
format_stamp: "Formatted at 2026-09-28 06:43:57 on dist-test-slave-b28r"
server_key: "045695a7373324a2af473eaaaec2e5de"
server_key_iv: "ba467501683e5efd2dcda13cf681fe0b"
server_key_version: "encryptionkey@0"
I20260928 06:43:57.418982 7097 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:43:57.419750 7112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:57.419904 7097 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:43:57.419983 7097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "011f174d16864b9e8c1c76e59686e04a"
format_stamp: "Formatted at 2026-09-28 06:43:57 on dist-test-slave-b28r"
server_key: "045695a7373324a2af473eaaaec2e5de"
server_key_iv: "ba467501683e5efd2dcda13cf681fe0b"
server_key_version: "encryptionkey@0"
I20260928 06:43:57.420040 7097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:43:57.430357 7097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:43:57.432179 7097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:43:57.432307 7097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:43:57.436138 7097 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38239
I20260928 06:43:57.436182 7164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38239 every 8 connection(s)
I20260928 06:43:57.436478 7097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:43:57.437254 7165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:43:57.439543 7165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 011f174d16864b9e8c1c76e59686e04a: Bootstrap starting.
I20260928 06:43:57.440073 7165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 011f174d16864b9e8c1c76e59686e04a: Neither blocks nor log segments found. Creating new log.
I20260928 06:43:57.440327 7165 log.cc:824] T 00000000000000000000000000000000 P 011f174d16864b9e8c1c76e59686e04a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:43:57.440953 7165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 011f174d16864b9e8c1c76e59686e04a: No bootstrap required, opened a new log
I20260928 06:43:57.442562 7165 raft_consensus.cc:348] T 00000000000000000000000000000000 P 011f174d16864b9e8c1c76e59686e04a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011f174d16864b9e8c1c76e59686e04a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38239 } }
I20260928 06:43:57.442698 7165 raft_consensus.cc:374] T 00000000000000000000000000000000 P 011f174d16864b9e8c1c76e59686e04a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:43:57.442718 7165 raft_consensus.cc:729] T 00000000000000000000000000000000 P 011f174d16864b9e8c1c76e59686e04a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 011f174d16864b9e8c1c76e59686e04a, State: Initialized, Role: FOLLOWER
I20260928 06:43:57.442843 7165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 011f174d16864b9e8c1c76e59686e04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011f174d16864b9e8c1c76e59686e04a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38239 } }
I20260928 06:43:57.442909 7165 raft_consensus.cc:388] T 00000000000000000000000000000000 P 011f174d16864b9e8c1c76e59686e04a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:43:57.442927 7165 raft_consensus.cc:482] T 00000000000000000000000000000000 P 011f174d16864b9e8c1c76e59686e04a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:43:57.442955 7165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 011f174d16864b9e8c1c76e59686e04a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:43:57.443490 7165 raft_consensus.cc:504] T 00000000000000000000000000000000 P 011f174d16864b9e8c1c76e59686e04a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011f174d16864b9e8c1c76e59686e04a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38239 } }
I20260928 06:43:57.443579 7165 leader_election.cc:302] T 00000000000000000000000000000000 P 011f174d16864b9e8c1c76e59686e04a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 011f174d16864b9e8c1c76e59686e04a; no voters:
I20260928 06:43:57.443728 7165 leader_election.cc:288] T 00000000000000000000000000000000 P 011f174d16864b9e8c1c76e59686e04a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:43:57.443773 7168 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 011f174d16864b9e8c1c76e59686e04a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:43:57.443935 7168 raft_consensus.cc:686] T 00000000000000000000000000000000 P 011f174d16864b9e8c1c76e59686e04a [term 1 LEADER]: Becoming Leader. State: Replica: 011f174d16864b9e8c1c76e59686e04a, State: Running, Role: LEADER
I20260928 06:43:57.443965 7165 sys_catalog.cc:564] T 00000000000000000000000000000000 P 011f174d16864b9e8c1c76e59686e04a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:43:57.444082 7168 consensus_queue.cc:237] T 00000000000000000000000000000000 P 011f174d16864b9e8c1c76e59686e04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011f174d16864b9e8c1c76e59686e04a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38239 } }
I20260928 06:43:57.444446 7170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 011f174d16864b9e8c1c76e59686e04a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 011f174d16864b9e8c1c76e59686e04a. Latest consensus state: current_term: 1 leader_uuid: "011f174d16864b9e8c1c76e59686e04a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011f174d16864b9e8c1c76e59686e04a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38239 } } }
I20260928 06:43:57.444532 7170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 011f174d16864b9e8c1c76e59686e04a [sys.catalog]: This master's current role is: LEADER
I20260928 06:43:57.444561 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 7097
I20260928 06:43:57.444564 7169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 011f174d16864b9e8c1c76e59686e04a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "011f174d16864b9e8c1c76e59686e04a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "011f174d16864b9e8c1c76e59686e04a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38239 } } }
I20260928 06:43:57.444635 7169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 011f174d16864b9e8c1c76e59686e04a [sys.catalog]: This master's current role is: LEADER
I20260928 06:43:57.444670 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:43:57.444851 19277 external_mini_cluster.cc:1428] Setting key 2e7cbf8d1d190e88856d148084e8cff4
I20260928 06:43:57.445668 7176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:43:57.446154 7176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:43:57.447669 7176 catalog_manager.cc:1347] Generated new cluster ID: ae371ce197dd48d4960731626923b71b
I20260928 06:43:57.447726 7176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:43:57.450018 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:57.445329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45885 (local address 127.0.0.1:38239)
0928 06:43:57.445592 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:57.445640 (+ 48us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:57.445692 (+ 52us) server_negotiation.cc:407] Connection header received
0928 06:43:57.445816 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:57.445823 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:57.445865 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:57.445978 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:57.446177 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:57.447366 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:57.448245 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:57.448666 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:57.448769 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:57.449419 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:57.449423 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:57.449424 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:43:57.449585 (+ 161us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:57.449615 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:43:57.449625 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:57.449692 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:57.449786 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 06:43:57.449816 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":72,"threads_started":1}
I20260928 06:43:57.450847 19277 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:43:57.454799 7185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:57.452139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55033 (local address 127.0.0.1:38239)
0928 06:43:57.452172 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:57.452173 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:57.452284 (+ 111us) server_negotiation.cc:407] Connection header received
0928 06:43:57.452325 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:57.452326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:57.452348 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:57.452395 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:57.452503 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:57.452982 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:57.453657 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:57.453881 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:57.453934 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:57.454520 (+ 586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:57.454521 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:57.454522 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:43:57.454526 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:57.454541 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:43:57.454545 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:57.454589 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:57.454713 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 06:43:57.454727 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 06:43:57.485013 7176 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:43:57.485736 7176 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:43:57.491782 7176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 011f174d16864b9e8c1c76e59686e04a: Generated new TSK 0
I20260928 06:43:57.491971 7176 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:43:57.494235 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 7097
W20260928 06:43:57.498322 7082 connection.cc:535] client connection to 127.0.0.1:38239 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38239): Connection reset by peer (error 104)
2026-09-28T06:43:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1238 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:43:57 dist-test-slave-b28r krb5kdc[7196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:43:57 dist-test-slave-b28r krb5kdc[7196](info): set up 2 sockets
Sep 28 06:43:57 dist-test-slave-b28r krb5kdc[7196](info): commencing operation
krb5kdc: starting...
W20260928 06:43:58.459853 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:43:58 dist-test-slave-b28r krb5kdc[7196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:43:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:43:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:43:58.582098 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39169
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:43:58.663700 7209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:43:58.663905 7209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:43:58.663928 7209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:43:58.665059 7209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:43:58.665100 7209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:43:58.665113 7209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:43:58.665124 7209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:43:58.665138 7209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:43:58.666635 7209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39169
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:43:58.666862 7209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:43:58.667064 7209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:43:58.671741 7217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:58.671749 7215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:58.671746 7214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:58.671983 7209 server_base.cc:1034] running on GCE node
I20260928 06:43:58.672240 7209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:43:58.672525 7209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:43:58.673681 7209 hybrid_clock.cc:648] HybridClock initialized: now 1790577838673669 us; error 30 us; skew 500 ppm
Sep 28 06:43:58 dist-test-slave-b28r krb5kdc[7196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577838, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:43:58.676144 7209 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:43:58.676636 7209 webserver.cc:460] Webserver started at http://127.0.0.1:38007/ using document root <none> and password file <none>
I20260928 06:43:58.676858 7209 fs_manager.cc:359] Metadata directory not provided
I20260928 06:43:58.676906 7209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:43:58.677030 7209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:43:58.678079 7209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "2e49a846b6cb48bab2ddf7fc42b76eaa"
format_stamp: "Formatted at 2026-09-28 06:43:58 on dist-test-slave-b28r"
server_key: "d5d0120b0e1e0d23593c8284ab876096"
server_key_iv: "fadfbb590d486f3734288f72d5f2a3b6"
server_key_version: "encryptionkey@0"
I20260928 06:43:58.678411 7209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "2e49a846b6cb48bab2ddf7fc42b76eaa"
format_stamp: "Formatted at 2026-09-28 06:43:58 on dist-test-slave-b28r"
server_key: "d5d0120b0e1e0d23593c8284ab876096"
server_key_iv: "fadfbb590d486f3734288f72d5f2a3b6"
server_key_version: "encryptionkey@0"
I20260928 06:43:58.679709 7209 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 06:43:58.680397 7224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:58.680545 7209 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:43:58.680619 7209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "2e49a846b6cb48bab2ddf7fc42b76eaa"
format_stamp: "Formatted at 2026-09-28 06:43:58 on dist-test-slave-b28r"
server_key: "d5d0120b0e1e0d23593c8284ab876096"
server_key_iv: "fadfbb590d486f3734288f72d5f2a3b6"
server_key_version: "encryptionkey@0"
I20260928 06:43:58.680680 7209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:43:58.683955 7209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:43:58.685695 7209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:43:58.685827 7209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:43:58.689610 7209 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39169
I20260928 06:43:58.689657 7276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39169 every 8 connection(s)
I20260928 06:43:58.689965 7209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:43:58.690770 7277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:43:58.693125 7277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e49a846b6cb48bab2ddf7fc42b76eaa: Bootstrap starting.
I20260928 06:43:58.693742 7277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e49a846b6cb48bab2ddf7fc42b76eaa: Neither blocks nor log segments found. Creating new log.
I20260928 06:43:58.693996 7277 log.cc:824] T 00000000000000000000000000000000 P 2e49a846b6cb48bab2ddf7fc42b76eaa: Log is configured to *not* fsync() on all Append() calls
I20260928 06:43:58.694621 7277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e49a846b6cb48bab2ddf7fc42b76eaa: No bootstrap required, opened a new log
I20260928 06:43:58.696300 7277 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e49a846b6cb48bab2ddf7fc42b76eaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e49a846b6cb48bab2ddf7fc42b76eaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39169 } }
I20260928 06:43:58.696439 7277 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e49a846b6cb48bab2ddf7fc42b76eaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:43:58.696460 7277 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e49a846b6cb48bab2ddf7fc42b76eaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e49a846b6cb48bab2ddf7fc42b76eaa, State: Initialized, Role: FOLLOWER
I20260928 06:43:58.696583 7277 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e49a846b6cb48bab2ddf7fc42b76eaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e49a846b6cb48bab2ddf7fc42b76eaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39169 } }
I20260928 06:43:58.696667 7277 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e49a846b6cb48bab2ddf7fc42b76eaa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:43:58.696700 7277 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e49a846b6cb48bab2ddf7fc42b76eaa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:43:58.696741 7277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e49a846b6cb48bab2ddf7fc42b76eaa [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:43:58.696789 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 7209
I20260928 06:43:58.696902 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:43:58.697080 19277 external_mini_cluster.cc:1428] Setting key fffa3821243427097316a8ae81ad4abc
I20260928 06:43:58.697392 7277 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e49a846b6cb48bab2ddf7fc42b76eaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e49a846b6cb48bab2ddf7fc42b76eaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39169 } }
I20260928 06:43:58.697499 7277 leader_election.cc:302] T 00000000000000000000000000000000 P 2e49a846b6cb48bab2ddf7fc42b76eaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e49a846b6cb48bab2ddf7fc42b76eaa; no voters:
I20260928 06:43:58.697666 7277 leader_election.cc:288] T 00000000000000000000000000000000 P 2e49a846b6cb48bab2ddf7fc42b76eaa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:43:58.697820 7283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e49a846b6cb48bab2ddf7fc42b76eaa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:43:58.697922 7277 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e49a846b6cb48bab2ddf7fc42b76eaa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:43:58.697939 7283 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e49a846b6cb48bab2ddf7fc42b76eaa [term 1 LEADER]: Becoming Leader. State: Replica: 2e49a846b6cb48bab2ddf7fc42b76eaa, State: Running, Role: LEADER
I20260928 06:43:58.699222 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:58.697548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58509 (local address 127.0.0.1:39169)
0928 06:43:58.697733 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:58.697743 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:58.697777 (+ 34us) server_negotiation.cc:407] Connection header received
0928 06:43:58.697899 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:58.697906 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:58.697938 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:58.698109 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:58.698767 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:58.698770 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:58.698771 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:43:58.698775 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:58.698815 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:43:58.698830 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:58.698885 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:58.698950 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 06:43:58.699003 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":94,"threads_started":1}
I20260928 06:43:58.699225 7283 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e49a846b6cb48bab2ddf7fc42b76eaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e49a846b6cb48bab2ddf7fc42b76eaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39169 } }
I20260928 06:43:58.699945 7284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e49a846b6cb48bab2ddf7fc42b76eaa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e49a846b6cb48bab2ddf7fc42b76eaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e49a846b6cb48bab2ddf7fc42b76eaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39169 } } }
I20260928 06:43:58.699828 7281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e49a846b6cb48bab2ddf7fc42b76eaa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e49a846b6cb48bab2ddf7fc42b76eaa. Latest consensus state: current_term: 1 leader_uuid: "2e49a846b6cb48bab2ddf7fc42b76eaa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e49a846b6cb48bab2ddf7fc42b76eaa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39169 } } }
I20260928 06:43:58.700016 7284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e49a846b6cb48bab2ddf7fc42b76eaa [sys.catalog]: This master's current role is: LEADER
I20260928 06:43:58.700021 7281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e49a846b6cb48bab2ddf7fc42b76eaa [sys.catalog]: This master's current role is: LEADER
W20260928 06:43:58.700328 7296 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2e49a846b6cb48bab2ddf7fc42b76eaa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:43:58.700379 7296 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:43:58.700461 7297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:43:58.700510 19277 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:43:58.700639 7297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:43:58.702005 7297 catalog_manager.cc:1347] Generated new cluster ID: 283b7ce07c0a47f5b2d4734d047c6eee
I20260928 06:43:58.702062 7297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:43:58.702733 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:58.701780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50069 (local address 127.0.0.1:39169)
0928 06:43:58.701814 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:58.701815 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:58.701930 (+ 115us) server_negotiation.cc:407] Connection header received
0928 06:43:58.701981 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:58.701981 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:58.701998 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:58.702042 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:58.702541 (+ 499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:58.702542 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:58.702542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:43:58.702546 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:58.702560 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:43:58.702563 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:58.702608 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:58.702656 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 06:43:58.702669 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:43:58.719653 7297 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:43:58.719878 7297 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:43:58.727135 7297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e49a846b6cb48bab2ddf7fc42b76eaa: Generated new TSK 0
I20260928 06:43:58.727361 7297 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:43:58.735288 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 7209
2026-09-28T06:43:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1241 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:43:58 dist-test-slave-b28r krb5kdc[7308](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:43:58 dist-test-slave-b28r krb5kdc[7308](info): set up 2 sockets
Sep 28 06:43:58 dist-test-slave-b28r krb5kdc[7308](info): commencing operation
krb5kdc: starting...
W20260928 06:43:59.679973 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:43:59 dist-test-slave-b28r krb5kdc[7308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:43:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:43:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:43:59.797680 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34051
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:43:59.876464 7321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:43:59.876672 7321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:43:59.876693 7321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:43:59.877831 7321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:43:59.877879 7321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:43:59.877892 7321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:43:59.877904 7321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:43:59.877914 7321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:43:59.879393 7321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:43:59.879621 7321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:43:59.879837 7321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:43:59.884405 7327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:43:59.884402 7326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:59.884543 7321 server_base.cc:1034] running on GCE node
W20260928 06:43:59.884486 7329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:43:59.884824 7321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:43:59.885064 7321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:43:59.886195 7321 hybrid_clock.cc:648] HybridClock initialized: now 1790577839886183 us; error 30 us; skew 500 ppm
Sep 28 06:43:59 dist-test-slave-b28r krb5kdc[7308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577839, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:43:59.888600 7321 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:43:59.889089 7321 webserver.cc:460] Webserver started at http://127.0.0.1:35909/ using document root <none> and password file <none>
I20260928 06:43:59.889371 7321 fs_manager.cc:359] Metadata directory not provided
I20260928 06:43:59.889431 7321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:43:59.889568 7321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:43:59.890870 7321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "4151ab39fc014f9f9fd2dc143936a5b2"
format_stamp: "Formatted at 2026-09-28 06:43:59 on dist-test-slave-b28r"
server_key: "cc07ebbca7adb26af8f5756917c9a78a"
server_key_iv: "cc90a0e10d0638fa44e19970cbad3594"
server_key_version: "encryptionkey@0"
I20260928 06:43:59.891284 7321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "4151ab39fc014f9f9fd2dc143936a5b2"
format_stamp: "Formatted at 2026-09-28 06:43:59 on dist-test-slave-b28r"
server_key: "cc07ebbca7adb26af8f5756917c9a78a"
server_key_iv: "cc90a0e10d0638fa44e19970cbad3594"
server_key_version: "encryptionkey@0"
I20260928 06:43:59.892920 7321 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:43:59.893709 7336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:43:59.893916 7321 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260928 06:43:59.894016 7321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "4151ab39fc014f9f9fd2dc143936a5b2"
format_stamp: "Formatted at 2026-09-28 06:43:59 on dist-test-slave-b28r"
server_key: "cc07ebbca7adb26af8f5756917c9a78a"
server_key_iv: "cc90a0e10d0638fa44e19970cbad3594"
server_key_version: "encryptionkey@0"
I20260928 06:43:59.894090 7321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:43:59.906121 7321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:43:59.908203 7321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:43:59.908315 7321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:43:59.912098 7321 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34051
I20260928 06:43:59.912149 7388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34051 every 8 connection(s)
I20260928 06:43:59.912461 7321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:43:59.913385 7389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:43:59.915856 7389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4151ab39fc014f9f9fd2dc143936a5b2: Bootstrap starting.
I20260928 06:43:59.916455 7389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4151ab39fc014f9f9fd2dc143936a5b2: Neither blocks nor log segments found. Creating new log.
I20260928 06:43:59.916729 7389 log.cc:824] T 00000000000000000000000000000000 P 4151ab39fc014f9f9fd2dc143936a5b2: Log is configured to *not* fsync() on all Append() calls
I20260928 06:43:59.917501 7389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4151ab39fc014f9f9fd2dc143936a5b2: No bootstrap required, opened a new log
I20260928 06:43:59.919104 7389 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4151ab39fc014f9f9fd2dc143936a5b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4151ab39fc014f9f9fd2dc143936a5b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34051 } }
I20260928 06:43:59.919240 7389 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4151ab39fc014f9f9fd2dc143936a5b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:43:59.919269 7389 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4151ab39fc014f9f9fd2dc143936a5b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4151ab39fc014f9f9fd2dc143936a5b2, State: Initialized, Role: FOLLOWER
I20260928 06:43:59.919395 7389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4151ab39fc014f9f9fd2dc143936a5b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4151ab39fc014f9f9fd2dc143936a5b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34051 } }
I20260928 06:43:59.919458 7389 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4151ab39fc014f9f9fd2dc143936a5b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:43:59.919478 7389 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4151ab39fc014f9f9fd2dc143936a5b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:43:59.919510 7389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4151ab39fc014f9f9fd2dc143936a5b2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:43:59.920006 7389 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4151ab39fc014f9f9fd2dc143936a5b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4151ab39fc014f9f9fd2dc143936a5b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34051 } }
I20260928 06:43:59.920096 7389 leader_election.cc:302] T 00000000000000000000000000000000 P 4151ab39fc014f9f9fd2dc143936a5b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4151ab39fc014f9f9fd2dc143936a5b2; no voters:
I20260928 06:43:59.920266 7389 leader_election.cc:288] T 00000000000000000000000000000000 P 4151ab39fc014f9f9fd2dc143936a5b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:43:59.920290 7392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4151ab39fc014f9f9fd2dc143936a5b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:43:59.920424 7392 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4151ab39fc014f9f9fd2dc143936a5b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4151ab39fc014f9f9fd2dc143936a5b2, State: Running, Role: LEADER
I20260928 06:43:59.920493 7389 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4151ab39fc014f9f9fd2dc143936a5b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:43:59.920534 7392 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4151ab39fc014f9f9fd2dc143936a5b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4151ab39fc014f9f9fd2dc143936a5b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34051 } }
I20260928 06:43:59.921051 7393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4151ab39fc014f9f9fd2dc143936a5b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4151ab39fc014f9f9fd2dc143936a5b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4151ab39fc014f9f9fd2dc143936a5b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34051 } } }
I20260928 06:43:59.921135 7393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4151ab39fc014f9f9fd2dc143936a5b2 [sys.catalog]: This master's current role is: LEADER
I20260928 06:43:59.921595 7401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:43:59.921671 7394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4151ab39fc014f9f9fd2dc143936a5b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4151ab39fc014f9f9fd2dc143936a5b2. Latest consensus state: current_term: 1 leader_uuid: "4151ab39fc014f9f9fd2dc143936a5b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4151ab39fc014f9f9fd2dc143936a5b2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34051 } } }
I20260928 06:43:59.921782 7394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4151ab39fc014f9f9fd2dc143936a5b2 [sys.catalog]: This master's current role is: LEADER
I20260928 06:43:59.922112 7401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:43:59.922202 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 7321
I20260928 06:43:59.922328 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:43:59.922518 19277 external_mini_cluster.cc:1428] Setting key e62dc1968d879840d2df5f433de38da0
I20260928 06:43:59.923619 7401 catalog_manager.cc:1347] Generated new cluster ID: dd4b228e71d64da6b074dde7d02609bf
I20260928 06:43:59.923668 7401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:43:59 dist-test-slave-b28r krb5kdc[7308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 06:43:59.933934 7411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:59.922942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57407 (local address 127.0.0.1:34051)
0928 06:43:59.923127 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:59.923137 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:59.923297 (+ 160us) server_negotiation.cc:407] Connection header received
0928 06:43:59.923415 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:59.923422 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:59.923476 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:59.923598 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:59.924476 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:59.925864 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:59.926969 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:59.927560 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:59.927661 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:59.930376 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:59.930380 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:59.930381 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:43:59.930393 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:59.932347 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:59.932703 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:59.932706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:59.932707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:59.932816 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:59.932983 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:59.932984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:59.932984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:59.933302 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:59.933451 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:59.933550 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 06:43:59.933687 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":83,"threads_started":1}
I20260928 06:43:59.934970 19277 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:43:59.943073 7411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:43:59.936367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43437 (local address 127.0.0.1:34051)
0928 06:43:59.936409 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:43:59.936410 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:43:59.936558 (+ 148us) server_negotiation.cc:407] Connection header received
0928 06:43:59.936600 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:43:59.936600 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:43:59.936609 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:43:59.936673 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:43:59.937213 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:59.937972 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:59.938941 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:43:59.939334 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:43:59.939402 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:43:59.940601 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:43:59.940602 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:43:59.940602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:43:59.940610 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:43:59.942049 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:59.942389 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:59.942390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:59.942390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:59.942444 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:43:59.942601 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:43:59.942602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:43:59.942602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:43:59.942720 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:43:59.942823 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 06:43:59.942901 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 06:43:59.942977 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:43:59.947754 7401 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:43:59.948472 7401 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:43:59.966698 7401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4151ab39fc014f9f9fd2dc143936a5b2: Generated new TSK 0
I20260928 06:43:59.966924 7401 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:43:59.968896 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 7321
2026-09-28T06:43:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1233 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:44:00 dist-test-slave-b28r krb5kdc[7420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:44:00 dist-test-slave-b28r krb5kdc[7420](info): set up 2 sockets
Sep 28 06:44:00 dist-test-slave-b28r krb5kdc[7420](info): commencing operation
krb5kdc: starting...
W20260928 06:44:00.915958 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:44:00 dist-test-slave-b28r krb5kdc[7420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577840, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:44:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:44:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:44:01.023970 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46105
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:44:01.103148 7433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:44:01.103372 7433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:44:01.103394 7433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:44:01.104553 7433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:44:01.104595 7433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:44:01.104609 7433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:44:01.104619 7433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:44:01.104629 7433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:44:01.106081 7433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:44:01.106318 7433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:44:01.106521 7433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:44:01.110934 7441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:44:01.110962 7439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:44:01.111021 7438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:44:01.111377 7433 server_base.cc:1034] running on GCE node
I20260928 06:44:01.111605 7433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:44:01.111879 7433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:44:01.113029 7433 hybrid_clock.cc:648] HybridClock initialized: now 1790577841113011 us; error 32 us; skew 500 ppm
Sep 28 06:44:01 dist-test-slave-b28r krb5kdc[7420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577841, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:44:01.115548 7433 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:44:01.115985 7433 webserver.cc:460] Webserver started at http://127.0.0.1:46867/ using document root <none> and password file <none>
I20260928 06:44:01.116197 7433 fs_manager.cc:359] Metadata directory not provided
I20260928 06:44:01.116246 7433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:44:01.116374 7433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:44:01.117486 7433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "41674c88b6ea4e69afb4cb406c0f0d0c"
format_stamp: "Formatted at 2026-09-28 06:44:01 on dist-test-slave-b28r"
server_key: "2b47f647929b4b5a3fc39d4510116586"
server_key_iv: "57d5a3d368bd76a32477d49b0aa821bc"
server_key_version: "encryptionkey@0"
I20260928 06:44:01.117815 7433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "41674c88b6ea4e69afb4cb406c0f0d0c"
format_stamp: "Formatted at 2026-09-28 06:44:01 on dist-test-slave-b28r"
server_key: "2b47f647929b4b5a3fc39d4510116586"
server_key_iv: "57d5a3d368bd76a32477d49b0aa821bc"
server_key_version: "encryptionkey@0"
I20260928 06:44:01.119232 7433 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:44:01.120024 7448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:44:01.120234 7433 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:44:01.120328 7433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "41674c88b6ea4e69afb4cb406c0f0d0c"
format_stamp: "Formatted at 2026-09-28 06:44:01 on dist-test-slave-b28r"
server_key: "2b47f647929b4b5a3fc39d4510116586"
server_key_iv: "57d5a3d368bd76a32477d49b0aa821bc"
server_key_version: "encryptionkey@0"
I20260928 06:44:01.120399 7433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:44:01.155041 7433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:44:01.157364 7433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:44:01.157539 7433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:44:01.161739 7433 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46105
I20260928 06:44:01.161813 7500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46105 every 8 connection(s)
I20260928 06:44:01.162130 7433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:44:01.162968 7501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:44:01.165275 7501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41674c88b6ea4e69afb4cb406c0f0d0c: Bootstrap starting.
I20260928 06:44:01.165840 7501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41674c88b6ea4e69afb4cb406c0f0d0c: Neither blocks nor log segments found. Creating new log.
I20260928 06:44:01.166097 7501 log.cc:824] T 00000000000000000000000000000000 P 41674c88b6ea4e69afb4cb406c0f0d0c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:44:01.166826 7501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41674c88b6ea4e69afb4cb406c0f0d0c: No bootstrap required, opened a new log
I20260928 06:44:01.168572 7501 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41674c88b6ea4e69afb4cb406c0f0d0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41674c88b6ea4e69afb4cb406c0f0d0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46105 } }
I20260928 06:44:01.168717 7501 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41674c88b6ea4e69afb4cb406c0f0d0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:44:01.168738 7501 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41674c88b6ea4e69afb4cb406c0f0d0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41674c88b6ea4e69afb4cb406c0f0d0c, State: Initialized, Role: FOLLOWER
I20260928 06:44:01.168737 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 7433
I20260928 06:44:01.168857 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:44:01.168865 7501 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41674c88b6ea4e69afb4cb406c0f0d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41674c88b6ea4e69afb4cb406c0f0d0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46105 } }
I20260928 06:44:01.168949 7501 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41674c88b6ea4e69afb4cb406c0f0d0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:44:01.168982 7501 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41674c88b6ea4e69afb4cb406c0f0d0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:44:01.169031 7501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41674c88b6ea4e69afb4cb406c0f0d0c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:44:01.169044 19277 external_mini_cluster.cc:1428] Setting key 016ddc6db8b1617015e9b76f3a3b4fac
I20260928 06:44:01.169711 7501 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41674c88b6ea4e69afb4cb406c0f0d0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41674c88b6ea4e69afb4cb406c0f0d0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46105 } }
I20260928 06:44:01.169811 7501 leader_election.cc:302] T 00000000000000000000000000000000 P 41674c88b6ea4e69afb4cb406c0f0d0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41674c88b6ea4e69afb4cb406c0f0d0c; no voters:
I20260928 06:44:01.169975 7501 leader_election.cc:288] T 00000000000000000000000000000000 P 41674c88b6ea4e69afb4cb406c0f0d0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:44:01.170018 7505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41674c88b6ea4e69afb4cb406c0f0d0c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:44:01.170161 7505 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41674c88b6ea4e69afb4cb406c0f0d0c [term 1 LEADER]: Becoming Leader. State: Replica: 41674c88b6ea4e69afb4cb406c0f0d0c, State: Running, Role: LEADER
I20260928 06:44:01.170311 7501 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41674c88b6ea4e69afb4cb406c0f0d0c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:44:01.170295 7505 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41674c88b6ea4e69afb4cb406c0f0d0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41674c88b6ea4e69afb4cb406c0f0d0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46105 } }
I20260928 06:44:01.170694 7507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41674c88b6ea4e69afb4cb406c0f0d0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41674c88b6ea4e69afb4cb406c0f0d0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41674c88b6ea4e69afb4cb406c0f0d0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46105 } } }
I20260928 06:44:01.170943 7507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41674c88b6ea4e69afb4cb406c0f0d0c [sys.catalog]: This master's current role is: LEADER
I20260928 06:44:01.170866 7508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41674c88b6ea4e69afb4cb406c0f0d0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41674c88b6ea4e69afb4cb406c0f0d0c. Latest consensus state: current_term: 1 leader_uuid: "41674c88b6ea4e69afb4cb406c0f0d0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41674c88b6ea4e69afb4cb406c0f0d0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46105 } } }
I20260928 06:44:01.171382 7508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41674c88b6ea4e69afb4cb406c0f0d0c [sys.catalog]: This master's current role is: LEADER
I20260928 06:44:01.171566 7515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:44:01.172115 7515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:44:01.173830 7515 catalog_manager.cc:1347] Generated new cluster ID: ca889884827f4ac290964a07def3874b
I20260928 06:44:01.173882 7515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:44:01.174952 7506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:01.169866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55923 (local address 127.0.0.1:46105)
0928 06:44:01.170067 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:01.170076 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:01.170113 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:44:01.170234 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:01.170241 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:01.170349 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:01.170494 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:01.170704 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:01.172108 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:01.172947 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:01.173518 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:01.173617 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:01.174271 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:01.174275 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:01.174276 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:44:01.174480 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:01.174521 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:44:01.174537 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:01.174614 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:01.174671 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 06:44:01.174741 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":106,"threads_started":1}
I20260928 06:44:01.175956 19277 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:44:01.180763 7506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:01.177395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52381 (local address 127.0.0.1:46105)
0928 06:44:01.177429 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:01.177430 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:01.177580 (+ 150us) server_negotiation.cc:407] Connection header received
0928 06:44:01.177631 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:01.177632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:01.177641 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:01.177687 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:01.177823 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:01.178510 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:01.179538 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:01.179850 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:01.179906 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:01.180547 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:01.180548 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:01.180549 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:44:01.180553 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:01.180567 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:44:01.180571 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:01.180617 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:01.180662 (+ 45us) server_negotiation.cc:299] Negotiation successful
0928 06:44:01.180687 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:44:01.188980 7515 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:44:01.189529 7515 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:44:01.202442 7515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41674c88b6ea4e69afb4cb406c0f0d0c: Generated new TSK 0
I20260928 06:44:01.202658 7515 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:44:01.205181 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 7433
2026-09-28T06:44:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1236 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:44:01 dist-test-slave-b28r krb5kdc[7532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:44:01 dist-test-slave-b28r krb5kdc[7532](info): set up 2 sockets
Sep 28 06:44:01 dist-test-slave-b28r krb5kdc[7532](info): commencing operation
krb5kdc: starting...
W20260928 06:44:02.160034 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:44:02 dist-test-slave-b28r krb5kdc[7532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:44:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:44:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:44:02.272316 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44601
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:44:02.352072 7545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:44:02.352268 7545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:44:02.352290 7545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:44:02.353410 7545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:44:02.353453 7545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:44:02.353466 7545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:44:02.353477 7545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:44:02.353488 7545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:44:02.355423 7545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:44:02.355719 7545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:44:02.355970 7545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:44:02.365869 7551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:44:02.365885 7550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:44:02.365885 7553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:44:02.366015 7545 server_base.cc:1034] running on GCE node
I20260928 06:44:02.366199 7545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:44:02.366497 7545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:44:02.367641 7545 hybrid_clock.cc:648] HybridClock initialized: now 1790577842367610 us; error 40 us; skew 500 ppm
Sep 28 06:44:02 dist-test-slave-b28r krb5kdc[7532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577842, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:44:02.370203 7545 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:44:02.370661 7545 webserver.cc:460] Webserver started at http://127.0.0.1:42347/ using document root <none> and password file <none>
I20260928 06:44:02.370874 7545 fs_manager.cc:359] Metadata directory not provided
I20260928 06:44:02.370923 7545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:44:02.371057 7545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:44:02.372174 7545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "cb581bfc98554a36a1bd7a2827bf3839"
format_stamp: "Formatted at 2026-09-28 06:44:02 on dist-test-slave-b28r"
server_key: "55dabd09bc2cdf74eab909e59977aa19"
server_key_iv: "3cefe43e55807c7182e1ddc2706320d8"
server_key_version: "encryptionkey@0"
I20260928 06:44:02.372509 7545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "cb581bfc98554a36a1bd7a2827bf3839"
format_stamp: "Formatted at 2026-09-28 06:44:02 on dist-test-slave-b28r"
server_key: "55dabd09bc2cdf74eab909e59977aa19"
server_key_iv: "3cefe43e55807c7182e1ddc2706320d8"
server_key_version: "encryptionkey@0"
I20260928 06:44:02.373801 7545 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:44:02.374568 7560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:44:02.374747 7545 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:44:02.374843 7545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "cb581bfc98554a36a1bd7a2827bf3839"
format_stamp: "Formatted at 2026-09-28 06:44:02 on dist-test-slave-b28r"
server_key: "55dabd09bc2cdf74eab909e59977aa19"
server_key_iv: "3cefe43e55807c7182e1ddc2706320d8"
server_key_version: "encryptionkey@0"
I20260928 06:44:02.374925 7545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:44:02.378330 7545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:44:02.380196 7545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:44:02.380386 7545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:44:02.384147 7545 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44601
I20260928 06:44:02.384186 7612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44601 every 8 connection(s)
I20260928 06:44:02.384480 7545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:44:02.385269 7613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:44:02.386932 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 7545
I20260928 06:44:02.387053 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:44:02.387282 19277 external_mini_cluster.cc:1428] Setting key 7ff097239606f55ec09323cfb35d8033
I20260928 06:44:02.387738 7613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb581bfc98554a36a1bd7a2827bf3839: Bootstrap starting.
I20260928 06:44:02.388294 7613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb581bfc98554a36a1bd7a2827bf3839: Neither blocks nor log segments found. Creating new log.
I20260928 06:44:02.388552 7613 log.cc:824] T 00000000000000000000000000000000 P cb581bfc98554a36a1bd7a2827bf3839: Log is configured to *not* fsync() on all Append() calls
I20260928 06:44:02.389251 7613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb581bfc98554a36a1bd7a2827bf3839: No bootstrap required, opened a new log
I20260928 06:44:02.389632 7616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:02.387753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51421 (local address 127.0.0.1:44601)
0928 06:44:02.387935 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:02.387945 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:02.387976 (+ 31us) server_negotiation.cc:407] Connection header received
0928 06:44:02.388431 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:02.388438 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:02.388464 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:02.388588 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:02.389138 (+ 550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:02.389142 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:02.389142 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:44:02.389146 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:02.389197 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:44:02.389207 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:02.389278 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:02.389444 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 06:44:02.389487 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":76,"threads_started":1}
I20260928 06:44:02.390471 19277 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:44:02.391420 7613 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb581bfc98554a36a1bd7a2827bf3839 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb581bfc98554a36a1bd7a2827bf3839" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44601 } }
I20260928 06:44:02.391594 7613 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb581bfc98554a36a1bd7a2827bf3839 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:44:02.391655 7613 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb581bfc98554a36a1bd7a2827bf3839 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb581bfc98554a36a1bd7a2827bf3839, State: Initialized, Role: FOLLOWER
I20260928 06:44:02.391783 7613 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb581bfc98554a36a1bd7a2827bf3839 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb581bfc98554a36a1bd7a2827bf3839" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44601 } }
I20260928 06:44:02.391878 7613 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb581bfc98554a36a1bd7a2827bf3839 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:44:02.391903 7613 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb581bfc98554a36a1bd7a2827bf3839 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:44:02.391949 7613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb581bfc98554a36a1bd7a2827bf3839 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:44:02.392434 7613 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb581bfc98554a36a1bd7a2827bf3839 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb581bfc98554a36a1bd7a2827bf3839" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44601 } }
I20260928 06:44:02.392521 7613 leader_election.cc:302] T 00000000000000000000000000000000 P cb581bfc98554a36a1bd7a2827bf3839 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb581bfc98554a36a1bd7a2827bf3839; no voters:
I20260928 06:44:02.392681 7613 leader_election.cc:288] T 00000000000000000000000000000000 P cb581bfc98554a36a1bd7a2827bf3839 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:44:02.392711 7623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb581bfc98554a36a1bd7a2827bf3839 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:44:02.392822 7623 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb581bfc98554a36a1bd7a2827bf3839 [term 1 LEADER]: Becoming Leader. State: Replica: cb581bfc98554a36a1bd7a2827bf3839, State: Running, Role: LEADER
I20260928 06:44:02.392899 7623 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb581bfc98554a36a1bd7a2827bf3839 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb581bfc98554a36a1bd7a2827bf3839" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44601 } }
I20260928 06:44:02.392926 7613 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb581bfc98554a36a1bd7a2827bf3839 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:44:02.392997 7616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:02.391954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48905 (local address 127.0.0.1:44601)
0928 06:44:02.391986 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:02.391988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:02.392153 (+ 165us) server_negotiation.cc:407] Connection header received
0928 06:44:02.392202 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:02.392203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:02.392208 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:02.392252 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:02.392779 (+ 527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:02.392780 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:02.392780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:44:02.392785 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:02.392798 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:44:02.392802 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:02.392830 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:02.392932 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 06:44:02.392943 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:44:02.393226 7623 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb581bfc98554a36a1bd7a2827bf3839 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb581bfc98554a36a1bd7a2827bf3839. Latest consensus state: current_term: 1 leader_uuid: "cb581bfc98554a36a1bd7a2827bf3839" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb581bfc98554a36a1bd7a2827bf3839" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44601 } } }
I20260928 06:44:02.393293 7623 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb581bfc98554a36a1bd7a2827bf3839 [sys.catalog]: This master's current role is: LEADER
I20260928 06:44:02.393648 7625 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb581bfc98554a36a1bd7a2827bf3839 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb581bfc98554a36a1bd7a2827bf3839" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb581bfc98554a36a1bd7a2827bf3839" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44601 } } }
I20260928 06:44:02.393723 7625 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb581bfc98554a36a1bd7a2827bf3839 [sys.catalog]: This master's current role is: LEADER
W20260928 06:44:02.394503 7639 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cb581bfc98554a36a1bd7a2827bf3839: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:44:02.394546 7639 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:44:02.394605 7628 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:44:02.394779 7628 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:44:02.396203 7628 catalog_manager.cc:1347] Generated new cluster ID: e2327af3310f44c7a442eed71266a790
I20260928 06:44:02.396257 7628 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:44:02.429831 7628 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:44:02.430053 7628 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:44:02.435694 7628 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb581bfc98554a36a1bd7a2827bf3839: Generated new TSK 0
I20260928 06:44:02.435882 7628 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:44:02.451099 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 7545
2026-09-28T06:44:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1245 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:44:02 dist-test-slave-b28r krb5kdc[7644](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:44:02 dist-test-slave-b28r krb5kdc[7644](info): set up 2 sockets
Sep 28 06:44:02 dist-test-slave-b28r krb5kdc[7644](info): commencing operation
krb5kdc: starting...
W20260928 06:44:03.399909 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:44:03 dist-test-slave-b28r krb5kdc[7644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:44:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:44:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:44:03.509260 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36835
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:44:03.589013 7657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:44:03.589224 7657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:44:03.589246 7657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:44:03.590337 7657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:44:03.590375 7657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:44:03.590389 7657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:44:03.590399 7657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:44:03.590409 7657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:44:03.591820 7657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:44:03.592011 7657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:44:03.592175 7657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:44:03.596920 7662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:44:03.596935 7665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:44:03.596966 7657 server_base.cc:1034] running on GCE node
W20260928 06:44:03.596926 7663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:44:03.597376 7657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:44:03.597620 7657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:44:03.598771 7657 hybrid_clock.cc:648] HybridClock initialized: now 1790577843598747 us; error 29 us; skew 500 ppm
Sep 28 06:44:03 dist-test-slave-b28r krb5kdc[7644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:44:03.601263 7657 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:44:03.601701 7657 webserver.cc:460] Webserver started at http://127.0.0.1:33189/ using document root <none> and password file <none>
I20260928 06:44:03.601900 7657 fs_manager.cc:359] Metadata directory not provided
I20260928 06:44:03.601945 7657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:44:03.602051 7657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:44:03.603075 7657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "514b130945b04bc8b47cc541d3a94ab9"
format_stamp: "Formatted at 2026-09-28 06:44:03 on dist-test-slave-b28r"
server_key: "fc08e469892d2354b1a3ad10e2e0b683"
server_key_iv: "7436befd3c3c2a1f4eb2c0561f692c0f"
server_key_version: "encryptionkey@0"
I20260928 06:44:03.603492 7657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "514b130945b04bc8b47cc541d3a94ab9"
format_stamp: "Formatted at 2026-09-28 06:44:03 on dist-test-slave-b28r"
server_key: "fc08e469892d2354b1a3ad10e2e0b683"
server_key_iv: "7436befd3c3c2a1f4eb2c0561f692c0f"
server_key_version: "encryptionkey@0"
I20260928 06:44:03.604792 7657 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:44:03.605557 7672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:44:03.605717 7657 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:44:03.605798 7657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "514b130945b04bc8b47cc541d3a94ab9"
format_stamp: "Formatted at 2026-09-28 06:44:03 on dist-test-slave-b28r"
server_key: "fc08e469892d2354b1a3ad10e2e0b683"
server_key_iv: "7436befd3c3c2a1f4eb2c0561f692c0f"
server_key_version: "encryptionkey@0"
I20260928 06:44:03.605851 7657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:44:03.623154 7657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:44:03.625319 7657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:44:03.625432 7657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:44:03.629253 7657 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36835
I20260928 06:44:03.629300 7724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36835 every 8 connection(s)
I20260928 06:44:03.629626 7657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:44:03.630456 7725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:44:03.632786 7725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 514b130945b04bc8b47cc541d3a94ab9: Bootstrap starting.
I20260928 06:44:03.633353 7725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 514b130945b04bc8b47cc541d3a94ab9: Neither blocks nor log segments found. Creating new log.
I20260928 06:44:03.633621 7725 log.cc:824] T 00000000000000000000000000000000 P 514b130945b04bc8b47cc541d3a94ab9: Log is configured to *not* fsync() on all Append() calls
I20260928 06:44:03.633925 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 7657
I20260928 06:44:03.634054 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:44:03.634241 19277 external_mini_cluster.cc:1428] Setting key d622ce43a307097e9b89873ac8ca9ca9
I20260928 06:44:03.634418 7725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 514b130945b04bc8b47cc541d3a94ab9: No bootstrap required, opened a new log
I20260928 06:44:03.636492 7725 raft_consensus.cc:348] T 00000000000000000000000000000000 P 514b130945b04bc8b47cc541d3a94ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "514b130945b04bc8b47cc541d3a94ab9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36835 } }
I20260928 06:44:03.636665 7725 raft_consensus.cc:374] T 00000000000000000000000000000000 P 514b130945b04bc8b47cc541d3a94ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:44:03.636710 7725 raft_consensus.cc:729] T 00000000000000000000000000000000 P 514b130945b04bc8b47cc541d3a94ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 514b130945b04bc8b47cc541d3a94ab9, State: Initialized, Role: FOLLOWER
I20260928 06:44:03.636840 7725 consensus_queue.cc:260] T 00000000000000000000000000000000 P 514b130945b04bc8b47cc541d3a94ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "514b130945b04bc8b47cc541d3a94ab9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36835 } }
I20260928 06:44:03.636920 7725 raft_consensus.cc:388] T 00000000000000000000000000000000 P 514b130945b04bc8b47cc541d3a94ab9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:44:03.636960 7725 raft_consensus.cc:482] T 00000000000000000000000000000000 P 514b130945b04bc8b47cc541d3a94ab9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:44:03.637007 7725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 514b130945b04bc8b47cc541d3a94ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:44:03.637701 7725 raft_consensus.cc:504] T 00000000000000000000000000000000 P 514b130945b04bc8b47cc541d3a94ab9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "514b130945b04bc8b47cc541d3a94ab9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36835 } }
I20260928 06:44:03.637809 7725 leader_election.cc:302] T 00000000000000000000000000000000 P 514b130945b04bc8b47cc541d3a94ab9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 514b130945b04bc8b47cc541d3a94ab9; no voters:
I20260928 06:44:03.637959 7725 leader_election.cc:288] T 00000000000000000000000000000000 P 514b130945b04bc8b47cc541d3a94ab9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:44:03.638002 7730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 514b130945b04bc8b47cc541d3a94ab9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:44:03.638159 7730 raft_consensus.cc:686] T 00000000000000000000000000000000 P 514b130945b04bc8b47cc541d3a94ab9 [term 1 LEADER]: Becoming Leader. State: Replica: 514b130945b04bc8b47cc541d3a94ab9, State: Running, Role: LEADER
I20260928 06:44:03.638185 7725 sys_catalog.cc:564] T 00000000000000000000000000000000 P 514b130945b04bc8b47cc541d3a94ab9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:44:03.638255 7730 consensus_queue.cc:237] T 00000000000000000000000000000000 P 514b130945b04bc8b47cc541d3a94ab9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "514b130945b04bc8b47cc541d3a94ab9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36835 } }
I20260928 06:44:03.638572 7730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 514b130945b04bc8b47cc541d3a94ab9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 514b130945b04bc8b47cc541d3a94ab9. Latest consensus state: current_term: 1 leader_uuid: "514b130945b04bc8b47cc541d3a94ab9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "514b130945b04bc8b47cc541d3a94ab9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36835 } } }
I20260928 06:44:03.638644 7730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 514b130945b04bc8b47cc541d3a94ab9 [sys.catalog]: This master's current role is: LEADER
I20260928 06:44:03.638984 7732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 514b130945b04bc8b47cc541d3a94ab9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "514b130945b04bc8b47cc541d3a94ab9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "514b130945b04bc8b47cc541d3a94ab9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36835 } } }
I20260928 06:44:03.639032 7736 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:44:03.639055 7732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 514b130945b04bc8b47cc541d3a94ab9 [sys.catalog]: This master's current role is: LEADER
I20260928 06:44:03.640410 7736 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:44:03.641930 7736 catalog_manager.cc:1347] Generated new cluster ID: ae3d00c3b1304472913b8c6c8e0e007d
I20260928 06:44:03.641983 7736 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:44:03 dist-test-slave-b28r krb5kdc[7644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 06:44:03.647418 7729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:03.634727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:52585 (local address 127.0.0.1:36835)
0928 06:44:03.634934 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:03.634944 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:03.634982 (+ 38us) server_negotiation.cc:407] Connection header received
0928 06:44:03.635733 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:03.635745 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:03.635821 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:03.635981 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:03.636572 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:03.638284 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:03.639362 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:03.640073 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:03.640176 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:03.643439 (+ 3263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:03.643443 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:03.643444 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:03.643456 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:03.645332 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:03.645749 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:03.645753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:03.645753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:03.645867 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:03.646127 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:03.646128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:03.646128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:03.646456 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:03.646975 (+ 519us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:03.647064 (+ 89us) server_negotiation.cc:299] Negotiation successful
0928 06:44:03.647196 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":101,"threads_started":1}
I20260928 06:44:03.648526 19277 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:44:03.649294 7736 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:44:03.649881 7736 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:44:03.651870 7736 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 514b130945b04bc8b47cc541d3a94ab9: Generated new TSK 0
I20260928 06:44:03.652036 7736 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:44:03.658015 7729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:03.650009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:55049 (local address 127.0.0.1:36835)
0928 06:44:03.650083 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:03.650085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:03.651271 (+ 1186us) server_negotiation.cc:407] Connection header received
0928 06:44:03.651366 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:03.651367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:03.651377 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:03.651404 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:03.652098 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:03.652903 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:03.654003 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:03.654221 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:03.654280 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:03.655688 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:03.655689 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:03.655690 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:03.655696 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:03.656917 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:03.657316 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:03.657317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:03.657317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:03.657381 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:03.657582 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:03.657582 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:03.657583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:03.657682 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:03.657756 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:03.657896 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 06:44:03.657946 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 06:44:03.659951 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 7657
W20260928 06:44:03.663926 7642 connection.cc:535] client connection to 127.0.0.1:36835 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36835): Connection reset by peer (error 104)
2026-09-28T06:44:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1208 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:44:03 dist-test-slave-b28r krb5kdc[7756](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:44:03 dist-test-slave-b28r krb5kdc[7756](info): set up 2 sockets
Sep 28 06:44:03 dist-test-slave-b28r krb5kdc[7756](info): commencing operation
krb5kdc: starting...
W20260928 06:44:04.608080 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:44:04 dist-test-slave-b28r krb5kdc[7756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577844, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:44:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:44:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:44:04.715567 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46589
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:44:04.795106 7769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:44:04.795356 7769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:44:04.795392 7769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:44:04.796591 7769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:44:04.796643 7769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:44:04.796665 7769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:44:04.796684 7769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:44:04.796702 7769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:44:04.798264 7769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:44:04.798519 7769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:44:04.798751 7769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:44:04.803475 7774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:44:04.803500 7775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:44:04.803503 7777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:44:04.803579 7769 server_base.cc:1034] running on GCE node
I20260928 06:44:04.803910 7769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:44:04.804261 7769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:44:04.805492 7769 hybrid_clock.cc:648] HybridClock initialized: now 1790577844805458 us; error 38 us; skew 500 ppm
Sep 28 06:44:04 dist-test-slave-b28r krb5kdc[7756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577844, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:44:04.807945 7769 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:44:04.808394 7769 webserver.cc:460] Webserver started at http://127.0.0.1:41857/ using document root <none> and password file <none>
I20260928 06:44:04.808612 7769 fs_manager.cc:359] Metadata directory not provided
I20260928 06:44:04.808662 7769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:44:04.808790 7769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:44:04.809849 7769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "f355360038b847f8870dc71a7dee3904"
format_stamp: "Formatted at 2026-09-28 06:44:04 on dist-test-slave-b28r"
server_key: "85366f2fc61369d7a26e7d9e76b37d0e"
server_key_iv: "869a0ff920bf376db5e93c6b829b964b"
server_key_version: "encryptionkey@0"
I20260928 06:44:04.810179 7769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "f355360038b847f8870dc71a7dee3904"
format_stamp: "Formatted at 2026-09-28 06:44:04 on dist-test-slave-b28r"
server_key: "85366f2fc61369d7a26e7d9e76b37d0e"
server_key_iv: "869a0ff920bf376db5e93c6b829b964b"
server_key_version: "encryptionkey@0"
I20260928 06:44:04.811532 7769 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:44:04.812238 7784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:44:04.812394 7769 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:44:04.812479 7769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "f355360038b847f8870dc71a7dee3904"
format_stamp: "Formatted at 2026-09-28 06:44:04 on dist-test-slave-b28r"
server_key: "85366f2fc61369d7a26e7d9e76b37d0e"
server_key_iv: "869a0ff920bf376db5e93c6b829b964b"
server_key_version: "encryptionkey@0"
I20260928 06:44:04.812537 7769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:44:04.827503 7769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:44:04.829552 7769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:44:04.829690 7769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:44:04.833580 7769 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46589
I20260928 06:44:04.833624 7836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46589 every 8 connection(s)
I20260928 06:44:04.833950 7769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:44:04.834774 7837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:44:04.836983 7837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f355360038b847f8870dc71a7dee3904: Bootstrap starting.
I20260928 06:44:04.837500 7837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f355360038b847f8870dc71a7dee3904: Neither blocks nor log segments found. Creating new log.
I20260928 06:44:04.837733 7837 log.cc:824] T 00000000000000000000000000000000 P f355360038b847f8870dc71a7dee3904: Log is configured to *not* fsync() on all Append() calls
I20260928 06:44:04.838352 7837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f355360038b847f8870dc71a7dee3904: No bootstrap required, opened a new log
I20260928 06:44:04.839766 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 7769
I20260928 06:44:04.839869 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:44:04.840059 19277 external_mini_cluster.cc:1428] Setting key af1c4505ec3943fd884457b45c995724
I20260928 06:44:04.840065 7837 raft_consensus.cc:348] T 00000000000000000000000000000000 P f355360038b847f8870dc71a7dee3904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f355360038b847f8870dc71a7dee3904" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46589 } }
I20260928 06:44:04.840233 7837 raft_consensus.cc:374] T 00000000000000000000000000000000 P f355360038b847f8870dc71a7dee3904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:44:04.840279 7837 raft_consensus.cc:729] T 00000000000000000000000000000000 P f355360038b847f8870dc71a7dee3904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f355360038b847f8870dc71a7dee3904, State: Initialized, Role: FOLLOWER
I20260928 06:44:04.840427 7837 consensus_queue.cc:260] T 00000000000000000000000000000000 P f355360038b847f8870dc71a7dee3904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f355360038b847f8870dc71a7dee3904" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46589 } }
I20260928 06:44:04.840516 7837 raft_consensus.cc:388] T 00000000000000000000000000000000 P f355360038b847f8870dc71a7dee3904 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:44:04.840581 7837 raft_consensus.cc:482] T 00000000000000000000000000000000 P f355360038b847f8870dc71a7dee3904 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:44:04.840634 7837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f355360038b847f8870dc71a7dee3904 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:44:04.841459 7837 raft_consensus.cc:504] T 00000000000000000000000000000000 P f355360038b847f8870dc71a7dee3904 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f355360038b847f8870dc71a7dee3904" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46589 } }
I20260928 06:44:04.841550 7837 leader_election.cc:302] T 00000000000000000000000000000000 P f355360038b847f8870dc71a7dee3904 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f355360038b847f8870dc71a7dee3904; no voters:
I20260928 06:44:04.841738 7837 leader_election.cc:288] T 00000000000000000000000000000000 P f355360038b847f8870dc71a7dee3904 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:44:04.841807 7842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f355360038b847f8870dc71a7dee3904 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:44:04.841961 7842 raft_consensus.cc:686] T 00000000000000000000000000000000 P f355360038b847f8870dc71a7dee3904 [term 1 LEADER]: Becoming Leader. State: Replica: f355360038b847f8870dc71a7dee3904, State: Running, Role: LEADER
I20260928 06:44:04.841976 7837 sys_catalog.cc:564] T 00000000000000000000000000000000 P f355360038b847f8870dc71a7dee3904 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:44:04.842043 7842 consensus_queue.cc:237] T 00000000000000000000000000000000 P f355360038b847f8870dc71a7dee3904 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f355360038b847f8870dc71a7dee3904" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46589 } }
I20260928 06:44:04.842406 7843 sys_catalog.cc:455] T 00000000000000000000000000000000 P f355360038b847f8870dc71a7dee3904 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f355360038b847f8870dc71a7dee3904" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f355360038b847f8870dc71a7dee3904" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46589 } } }
I20260928 06:44:04.842489 7843 sys_catalog.cc:458] T 00000000000000000000000000000000 P f355360038b847f8870dc71a7dee3904 [sys.catalog]: This master's current role is: LEADER
I20260928 06:44:04.842622 7844 sys_catalog.cc:455] T 00000000000000000000000000000000 P f355360038b847f8870dc71a7dee3904 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f355360038b847f8870dc71a7dee3904. Latest consensus state: current_term: 1 leader_uuid: "f355360038b847f8870dc71a7dee3904" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f355360038b847f8870dc71a7dee3904" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46589 } } }
I20260928 06:44:04.842681 7844 sys_catalog.cc:458] T 00000000000000000000000000000000 P f355360038b847f8870dc71a7dee3904 [sys.catalog]: This master's current role is: LEADER
I20260928 06:44:04.843267 7850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:44:04.843820 7850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:44:04.845340 7850 catalog_manager.cc:1347] Generated new cluster ID: 80936079811e4297bd0d2bfa5fddbcc1
I20260928 06:44:04.845392 7850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:44:04.848552 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:04.840586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:47957 (local address 127.0.0.1:46589)
0928 06:44:04.840768 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:04.840777 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:04.843327 (+ 2550us) server_negotiation.cc:407] Connection header received
0928 06:44:04.843563 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:04.843572 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:04.843622 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:04.843705 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:04.844014 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:04.845187 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:04.846335 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:04.847003 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:04.847184 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:04.847871 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:04.847875 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:04.847875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:44:04.848096 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:04.848139 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:44:04.848155 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:04.848245 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:04.848315 (+ 70us) server_negotiation.cc:299] Negotiation successful
0928 06:44:04.848345 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":100,"threads_started":1}
I20260928 06:44:04.849479 19277 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:44:04.854314 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:04.850996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:36519 (local address 127.0.0.1:46589)
0928 06:44:04.851029 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:04.851030 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:04.851120 (+ 90us) server_negotiation.cc:407] Connection header received
0928 06:44:04.851222 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:04.851223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:04.851234 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:04.851301 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:04.851421 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:04.852176 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:04.853090 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:04.853395 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:04.853466 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:04.854086 (+ 620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:04.854087 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:04.854087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:44:04.854091 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:04.854106 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:44:04.854111 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:04.854171 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:04.854227 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 06:44:04.854239 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:44:04.858417 7850 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:44:04.858872 7850 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:44:04.863641 7850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f355360038b847f8870dc71a7dee3904: Generated new TSK 0
I20260928 06:44:04.863806 7850 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:44:04.866565 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 7769
W20260928 06:44:04.870680 7754 connection.cc:535] client connection to 127.0.0.1:46589 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46589): Connection reset by peer (error 104)
2026-09-28T06:44:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1206 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:44:04 dist-test-slave-b28r krb5kdc[7868](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:44:04 dist-test-slave-b28r krb5kdc[7868](info): set up 2 sockets
Sep 28 06:44:04 dist-test-slave-b28r krb5kdc[7868](info): commencing operation
krb5kdc: starting...
W20260928 06:44:05.812002 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:44:05 dist-test-slave-b28r krb5kdc[7868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:44:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:44:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:44:05.924223 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39451
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:44:06.003350 7881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:44:06.003554 7881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:44:06.003576 7881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:44:06.004724 7881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:44:06.004770 7881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:44:06.004782 7881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:44:06.004794 7881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:44:06.004805 7881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:44:06.006335 7881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:44:06.006572 7881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:44:06.006830 7881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:44:06.011461 7887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:44:06.011451 7889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:44:06.011451 7886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:44:06.011934 7881 server_base.cc:1034] running on GCE node
I20260928 06:44:06.012152 7881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:44:06.012449 7881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:44:06.013592 7881 hybrid_clock.cc:648] HybridClock initialized: now 1790577846013573 us; error 31 us; skew 500 ppm
Sep 28 06:44:06 dist-test-slave-b28r krb5kdc[7868](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577846, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:44:06.016167 7881 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:44:06.016615 7881 webserver.cc:460] Webserver started at http://127.0.0.1:40089/ using document root <none> and password file <none>
I20260928 06:44:06.016852 7881 fs_manager.cc:359] Metadata directory not provided
I20260928 06:44:06.016902 7881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:44:06.017038 7881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:44:06.018150 7881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "c84c44a85d9847ab957630c46d4d1b8a"
format_stamp: "Formatted at 2026-09-28 06:44:06 on dist-test-slave-b28r"
server_key: "6abd17b43e8bf33476fd0b859a411c0b"
server_key_iv: "fde107ec68e37dce84f2fbf559ef79fb"
server_key_version: "encryptionkey@0"
I20260928 06:44:06.018512 7881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "c84c44a85d9847ab957630c46d4d1b8a"
format_stamp: "Formatted at 2026-09-28 06:44:06 on dist-test-slave-b28r"
server_key: "6abd17b43e8bf33476fd0b859a411c0b"
server_key_iv: "fde107ec68e37dce84f2fbf559ef79fb"
server_key_version: "encryptionkey@0"
I20260928 06:44:06.020049 7881 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:44:06.020975 7896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:44:06.021147 7881 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:44:06.021268 7881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "c84c44a85d9847ab957630c46d4d1b8a"
format_stamp: "Formatted at 2026-09-28 06:44:06 on dist-test-slave-b28r"
server_key: "6abd17b43e8bf33476fd0b859a411c0b"
server_key_iv: "fde107ec68e37dce84f2fbf559ef79fb"
server_key_version: "encryptionkey@0"
I20260928 06:44:06.021330 7881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:44:06.024601 7881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:44:06.026541 7881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:44:06.026645 7881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:44:06.030401 7881 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39451
I20260928 06:44:06.030433 7948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39451 every 8 connection(s)
I20260928 06:44:06.030722 7881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:44:06.031566 7949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:44:06.033806 7949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c84c44a85d9847ab957630c46d4d1b8a: Bootstrap starting.
I20260928 06:44:06.034413 7949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c84c44a85d9847ab957630c46d4d1b8a: Neither blocks nor log segments found. Creating new log.
I20260928 06:44:06.034672 7949 log.cc:824] T 00000000000000000000000000000000 P c84c44a85d9847ab957630c46d4d1b8a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:44:06.035356 7949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c84c44a85d9847ab957630c46d4d1b8a: No bootstrap required, opened a new log
I20260928 06:44:06.036916 7949 raft_consensus.cc:348] T 00000000000000000000000000000000 P c84c44a85d9847ab957630c46d4d1b8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c84c44a85d9847ab957630c46d4d1b8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39451 } }
I20260928 06:44:06.037050 7949 raft_consensus.cc:374] T 00000000000000000000000000000000 P c84c44a85d9847ab957630c46d4d1b8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:44:06.037082 7949 raft_consensus.cc:729] T 00000000000000000000000000000000 P c84c44a85d9847ab957630c46d4d1b8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c84c44a85d9847ab957630c46d4d1b8a, State: Initialized, Role: FOLLOWER
I20260928 06:44:06.037209 7949 consensus_queue.cc:260] T 00000000000000000000000000000000 P c84c44a85d9847ab957630c46d4d1b8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c84c44a85d9847ab957630c46d4d1b8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39451 } }
I20260928 06:44:06.037267 7949 raft_consensus.cc:388] T 00000000000000000000000000000000 P c84c44a85d9847ab957630c46d4d1b8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:44:06.037297 7949 raft_consensus.cc:482] T 00000000000000000000000000000000 P c84c44a85d9847ab957630c46d4d1b8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:44:06.037343 7949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c84c44a85d9847ab957630c46d4d1b8a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:44:06.037839 7949 raft_consensus.cc:504] T 00000000000000000000000000000000 P c84c44a85d9847ab957630c46d4d1b8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c84c44a85d9847ab957630c46d4d1b8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39451 } }
I20260928 06:44:06.037932 7949 leader_election.cc:302] T 00000000000000000000000000000000 P c84c44a85d9847ab957630c46d4d1b8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c84c44a85d9847ab957630c46d4d1b8a; no voters:
I20260928 06:44:06.038081 7949 leader_election.cc:288] T 00000000000000000000000000000000 P c84c44a85d9847ab957630c46d4d1b8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:44:06.038120 7952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c84c44a85d9847ab957630c46d4d1b8a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:44:06.038283 7952 raft_consensus.cc:686] T 00000000000000000000000000000000 P c84c44a85d9847ab957630c46d4d1b8a [term 1 LEADER]: Becoming Leader. State: Replica: c84c44a85d9847ab957630c46d4d1b8a, State: Running, Role: LEADER
I20260928 06:44:06.038349 7949 sys_catalog.cc:564] T 00000000000000000000000000000000 P c84c44a85d9847ab957630c46d4d1b8a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:44:06.038375 7952 consensus_queue.cc:237] T 00000000000000000000000000000000 P c84c44a85d9847ab957630c46d4d1b8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c84c44a85d9847ab957630c46d4d1b8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39451 } }
I20260928 06:44:06.038569 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 7881
I20260928 06:44:06.038681 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:44:06.038762 7954 sys_catalog.cc:455] T 00000000000000000000000000000000 P c84c44a85d9847ab957630c46d4d1b8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c84c44a85d9847ab957630c46d4d1b8a. Latest consensus state: current_term: 1 leader_uuid: "c84c44a85d9847ab957630c46d4d1b8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c84c44a85d9847ab957630c46d4d1b8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39451 } } }
I20260928 06:44:06.038878 19277 external_mini_cluster.cc:1428] Setting key 40973d9e14a1d91e5cd721afb06b3621
I20260928 06:44:06.038846 7953 sys_catalog.cc:455] T 00000000000000000000000000000000 P c84c44a85d9847ab957630c46d4d1b8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c84c44a85d9847ab957630c46d4d1b8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c84c44a85d9847ab957630c46d4d1b8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39451 } } }
I20260928 06:44:06.038894 7954 sys_catalog.cc:458] T 00000000000000000000000000000000 P c84c44a85d9847ab957630c46d4d1b8a [sys.catalog]: This master's current role is: LEADER
I20260928 06:44:06.038915 7953 sys_catalog.cc:458] T 00000000000000000000000000000000 P c84c44a85d9847ab957630c46d4d1b8a [sys.catalog]: This master's current role is: LEADER
I20260928 06:44:06.039340 7956 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:44:06.039804 7956 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:44:06.042234 7956 catalog_manager.cc:1347] Generated new cluster ID: d37a9f70744a4fd29c4cffb76304a685
I20260928 06:44:06.042290 7956 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:44:06.043771 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:06.039587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:38385 (local address 127.0.0.1:39451)
0928 06:44:06.042303 (+ 2716us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:06.042311 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:06.042346 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:44:06.042456 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:06.042461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:06.042484 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:06.042612 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:06.043342 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:06.043345 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:06.043346 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:44:06.043351 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:06.043392 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:44:06.043408 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:06.043461 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:06.043528 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 06:44:06.043569 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2655,"thread_start_us":97,"threads_started":1}
I20260928 06:44:06.044584 19277 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:44:06.046746 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:06.045819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:45161 (local address 127.0.0.1:39451)
0928 06:44:06.045860 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:06.045862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:06.045973 (+ 111us) server_negotiation.cc:407] Connection header received
0928 06:44:06.046025 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:06.046026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:06.046031 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:06.046081 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:06.046575 (+ 494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:06.046576 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:06.046576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:44:06.046580 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:06.046592 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:44:06.046595 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:06.046639 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:06.046678 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 06:44:06.046690 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:44:06.048058 7956 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:44:06.048207 7956 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:44:06.056960 7956 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c84c44a85d9847ab957630c46d4d1b8a: Generated new TSK 0
I20260928 06:44:06.057128 7956 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:44:06.061618 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 7881
2026-09-28T06:44:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1194 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:44:06 dist-test-slave-b28r krb5kdc[7980](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:44:06 dist-test-slave-b28r krb5kdc[7980](info): set up 2 sockets
Sep 28 06:44:06 dist-test-slave-b28r krb5kdc[7980](info): commencing operation
krb5kdc: starting...
W20260928 06:44:07.003983 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:44:07 dist-test-slave-b28r krb5kdc[7980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:44:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:44:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:44:07.114362 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39525
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:44:07.192850 7993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:44:07.193060 7993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:44:07.193091 7993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:44:07.194224 7993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:44:07.194270 7993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:44:07.194288 7993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:44:07.194304 7993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:44:07.194319 7993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:44:07.195809 7993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39525
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:44:07.196048 7993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:44:07.196252 7993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:44:07.200905 7998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:44:07.201001 7993 server_base.cc:1034] running on GCE node
W20260928 06:44:07.200933 8001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:44:07.200933 7999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:44:07.201414 7993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:44:07.201704 7993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:44:07.202864 7993 hybrid_clock.cc:648] HybridClock initialized: now 1790577847202850 us; error 29 us; skew 500 ppm
Sep 28 06:44:07 dist-test-slave-b28r krb5kdc[7980](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577847, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:44:07.205467 7993 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:44:07.205907 7993 webserver.cc:460] Webserver started at http://127.0.0.1:39705/ using document root <none> and password file <none>
I20260928 06:44:07.206105 7993 fs_manager.cc:359] Metadata directory not provided
I20260928 06:44:07.206152 7993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:44:07.206280 7993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:44:07.207394 7993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "25efd7075b6846bda73ac91cffc74801"
format_stamp: "Formatted at 2026-09-28 06:44:07 on dist-test-slave-b28r"
server_key: "378c29765c88fce032903df440e1ef7d"
server_key_iv: "be604075b53c2007bb566d2ec6452298"
server_key_version: "encryptionkey@0"
I20260928 06:44:07.207731 7993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "25efd7075b6846bda73ac91cffc74801"
format_stamp: "Formatted at 2026-09-28 06:44:07 on dist-test-slave-b28r"
server_key: "378c29765c88fce032903df440e1ef7d"
server_key_iv: "be604075b53c2007bb566d2ec6452298"
server_key_version: "encryptionkey@0"
I20260928 06:44:07.209054 7993 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:44:07.209847 8008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:44:07.210012 7993 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:44:07.210095 7993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "25efd7075b6846bda73ac91cffc74801"
format_stamp: "Formatted at 2026-09-28 06:44:07 on dist-test-slave-b28r"
server_key: "378c29765c88fce032903df440e1ef7d"
server_key_iv: "be604075b53c2007bb566d2ec6452298"
server_key_version: "encryptionkey@0"
I20260928 06:44:07.210153 7993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:44:07.221078 7993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:44:07.223006 7993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:44:07.223155 7993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:44:07.226842 7993 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39525
I20260928 06:44:07.226895 8060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39525 every 8 connection(s)
I20260928 06:44:07.227285 7993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:44:07.228117 8061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:44:07.228709 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 7993
I20260928 06:44:07.228819 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:44:07.228994 19277 external_mini_cluster.cc:1428] Setting key 1da6035c76a2d6ca18ba17de6acbc557
I20260928 06:44:07.230923 8061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25efd7075b6846bda73ac91cffc74801: Bootstrap starting.
I20260928 06:44:07.231578 8061 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25efd7075b6846bda73ac91cffc74801: Neither blocks nor log segments found. Creating new log.
I20260928 06:44:07.231846 8061 log.cc:824] T 00000000000000000000000000000000 P 25efd7075b6846bda73ac91cffc74801: Log is configured to *not* fsync() on all Append() calls
I20260928 06:44:07.232502 8061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25efd7075b6846bda73ac91cffc74801: No bootstrap required, opened a new log
I20260928 06:44:07.234609 8061 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25efd7075b6846bda73ac91cffc74801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25efd7075b6846bda73ac91cffc74801" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39525 } }
I20260928 06:44:07.234752 8061 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25efd7075b6846bda73ac91cffc74801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:44:07.234774 8061 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25efd7075b6846bda73ac91cffc74801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25efd7075b6846bda73ac91cffc74801, State: Initialized, Role: FOLLOWER
I20260928 06:44:07.234882 8061 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25efd7075b6846bda73ac91cffc74801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25efd7075b6846bda73ac91cffc74801" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39525 } }
I20260928 06:44:07.234946 8061 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25efd7075b6846bda73ac91cffc74801 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:44:07.234967 8061 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25efd7075b6846bda73ac91cffc74801 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:44:07.235013 8061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25efd7075b6846bda73ac91cffc74801 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 06:44:07 dist-test-slave-b28r krb5kdc[7980](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 06:44:07.235711 8061 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25efd7075b6846bda73ac91cffc74801 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25efd7075b6846bda73ac91cffc74801" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39525 } }
I20260928 06:44:07.235838 8061 leader_election.cc:302] T 00000000000000000000000000000000 P 25efd7075b6846bda73ac91cffc74801 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25efd7075b6846bda73ac91cffc74801; no voters:
I20260928 06:44:07.236006 8061 leader_election.cc:288] T 00000000000000000000000000000000 P 25efd7075b6846bda73ac91cffc74801 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:44:07.236059 8066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25efd7075b6846bda73ac91cffc74801 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:44:07.236205 8066 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25efd7075b6846bda73ac91cffc74801 [term 1 LEADER]: Becoming Leader. State: Replica: 25efd7075b6846bda73ac91cffc74801, State: Running, Role: LEADER
I20260928 06:44:07.236267 8061 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25efd7075b6846bda73ac91cffc74801 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:44:07.236374 8066 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25efd7075b6846bda73ac91cffc74801 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25efd7075b6846bda73ac91cffc74801" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39525 } }
I20260928 06:44:07.236836 8068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25efd7075b6846bda73ac91cffc74801 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25efd7075b6846bda73ac91cffc74801. Latest consensus state: current_term: 1 leader_uuid: "25efd7075b6846bda73ac91cffc74801" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25efd7075b6846bda73ac91cffc74801" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39525 } } }
I20260928 06:44:07.237056 8068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25efd7075b6846bda73ac91cffc74801 [sys.catalog]: This master's current role is: LEADER
I20260928 06:44:07.236992 8067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25efd7075b6846bda73ac91cffc74801 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25efd7075b6846bda73ac91cffc74801" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25efd7075b6846bda73ac91cffc74801" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39525 } } }
I20260928 06:44:07.237588 8080 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:44:07.238281 8080 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:44:07.237607 8067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25efd7075b6846bda73ac91cffc74801 [sys.catalog]: This master's current role is: LEADER
I20260928 06:44:07.239876 8064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:07.229477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:48997 (local address 127.0.0.1:39525)
0928 06:44:07.229652 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:07.229662 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:07.229687 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:44:07.230087 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:07.230094 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:07.230139 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:07.230266 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:07.231072 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:07.232098 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:07.232908 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:07.233550 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:07.233665 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:07.236260 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:07.236264 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:07.236265 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:07.236277 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:07.238205 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:07.238614 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:07.238617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:07.238617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:07.238700 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:07.238928 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:07.238929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:07.238929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:07.239245 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:07.239439 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:07.239566 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 06:44:07.239663 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":72,"threads_started":1}
I20260928 06:44:07.240317 8080 catalog_manager.cc:1347] Generated new cluster ID: a7b67c9652e8476c8af5f522e88b4ec0
I20260928 06:44:07.240370 8080 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:44:07.240726 19277 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:44:07.248842 8064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:07.242034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:46555 (local address 127.0.0.1:39525)
0928 06:44:07.242064 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:07.242065 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:07.242164 (+ 99us) server_negotiation.cc:407] Connection header received
0928 06:44:07.242246 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:07.242247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:07.242255 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:07.242279 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:07.242874 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:07.243632 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:07.244573 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:07.244870 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:07.244903 (+ 33us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:07.246242 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:07.246243 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:07.246244 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:07.246250 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:07.247657 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:07.248061 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:07.248062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:07.248062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:07.248116 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:07.248329 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:07.248330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:07.248330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:07.248438 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:07.248502 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:07.248697 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 06:44:07.248755 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 06:44:07.254853 8080 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:44:07.255420 8080 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:44:07.261878 8080 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25efd7075b6846bda73ac91cffc74801: Generated new TSK 0
I20260928 06:44:07.262065 8080 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:44:07.268270 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 7993
W20260928 06:44:07.272289 7978 connection.cc:535] client connection to 127.0.0.1:39525 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39525): Connection reset by peer (error 104)
2026-09-28T06:44:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1206 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:44:07 dist-test-slave-b28r krb5kdc[8092](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:44:07 dist-test-slave-b28r krb5kdc[8092](info): set up 2 sockets
Sep 28 06:44:07 dist-test-slave-b28r krb5kdc[8092](info): commencing operation
krb5kdc: starting...
W20260928 06:44:08.216038 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:44:08 dist-test-slave-b28r krb5kdc[8092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:44:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:44:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:44:08.328013 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43457
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:44:08.408089 8105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:44:08.408295 8105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:44:08.408317 8105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:44:08.409435 8105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:44:08.409477 8105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:44:08.409489 8105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:44:08.409500 8105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:44:08.409518 8105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:44:08.410938 8105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43457
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:44:08.411183 8105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:44:08.411394 8105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:44:08.415968 8110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:44:08.416030 8111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:44:08.416141 8105 server_base.cc:1034] running on GCE node
W20260928 06:44:08.416235 8113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:44:08.416507 8105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:44:08.416814 8105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:44:08.417951 8105 hybrid_clock.cc:648] HybridClock initialized: now 1790577848417939 us; error 25 us; skew 500 ppm
Sep 28 06:44:08 dist-test-slave-b28r krb5kdc[8092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577848, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:44:08.425410 8105 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:44:08.425864 8105 webserver.cc:460] Webserver started at http://127.0.0.1:37451/ using document root <none> and password file <none>
I20260928 06:44:08.426082 8105 fs_manager.cc:359] Metadata directory not provided
I20260928 06:44:08.426131 8105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:44:08.426254 8105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:44:08.427362 8105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "7fe9ed18cf0e4a23b3744a8270d8dd48"
format_stamp: "Formatted at 2026-09-28 06:44:08 on dist-test-slave-b28r"
server_key: "0637b465ac037ef81892f6d7a270d505"
server_key_iv: "c88c88816304e37056c8504173fa4912"
server_key_version: "encryptionkey@0"
I20260928 06:44:08.427688 8105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "7fe9ed18cf0e4a23b3744a8270d8dd48"
format_stamp: "Formatted at 2026-09-28 06:44:08 on dist-test-slave-b28r"
server_key: "0637b465ac037ef81892f6d7a270d505"
server_key_iv: "c88c88816304e37056c8504173fa4912"
server_key_version: "encryptionkey@0"
I20260928 06:44:08.428936 8105 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:44:08.429641 8120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:44:08.429783 8105 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:44:08.429860 8105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "7fe9ed18cf0e4a23b3744a8270d8dd48"
format_stamp: "Formatted at 2026-09-28 06:44:08 on dist-test-slave-b28r"
server_key: "0637b465ac037ef81892f6d7a270d505"
server_key_iv: "c88c88816304e37056c8504173fa4912"
server_key_version: "encryptionkey@0"
I20260928 06:44:08.429920 8105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:44:08.441743 8105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:44:08.443578 8105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:44:08.443713 8105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:44:08.447634 8105 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43457
I20260928 06:44:08.447705 8172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43457 every 8 connection(s)
I20260928 06:44:08.448024 8105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:44:08.448820 8173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:44:08.451042 8173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fe9ed18cf0e4a23b3744a8270d8dd48: Bootstrap starting.
I20260928 06:44:08.451620 8173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fe9ed18cf0e4a23b3744a8270d8dd48: Neither blocks nor log segments found. Creating new log.
I20260928 06:44:08.451858 8173 log.cc:824] T 00000000000000000000000000000000 P 7fe9ed18cf0e4a23b3744a8270d8dd48: Log is configured to *not* fsync() on all Append() calls
I20260928 06:44:08.452509 8173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fe9ed18cf0e4a23b3744a8270d8dd48: No bootstrap required, opened a new log
I20260928 06:44:08.453094 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 8105
I20260928 06:44:08.453203 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:44:08.453372 19277 external_mini_cluster.cc:1428] Setting key 2c1d9e4f862954d232b8dcfd885aff2f
I20260928 06:44:08.454488 8173 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fe9ed18cf0e4a23b3744a8270d8dd48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe9ed18cf0e4a23b3744a8270d8dd48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43457 } }
I20260928 06:44:08.454661 8173 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fe9ed18cf0e4a23b3744a8270d8dd48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:44:08.454698 8173 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fe9ed18cf0e4a23b3744a8270d8dd48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fe9ed18cf0e4a23b3744a8270d8dd48, State: Initialized, Role: FOLLOWER
I20260928 06:44:08.454854 8173 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fe9ed18cf0e4a23b3744a8270d8dd48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe9ed18cf0e4a23b3744a8270d8dd48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43457 } }
I20260928 06:44:08.454931 8173 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fe9ed18cf0e4a23b3744a8270d8dd48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:44:08.454952 8173 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fe9ed18cf0e4a23b3744a8270d8dd48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:44:08.454986 8173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fe9ed18cf0e4a23b3744a8270d8dd48 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:44:08.455554 8173 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fe9ed18cf0e4a23b3744a8270d8dd48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe9ed18cf0e4a23b3744a8270d8dd48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43457 } }
I20260928 06:44:08.455638 8173 leader_election.cc:302] T 00000000000000000000000000000000 P 7fe9ed18cf0e4a23b3744a8270d8dd48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fe9ed18cf0e4a23b3744a8270d8dd48; no voters:
I20260928 06:44:08.455785 8173 leader_election.cc:288] T 00000000000000000000000000000000 P 7fe9ed18cf0e4a23b3744a8270d8dd48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:44:08.455809 8178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fe9ed18cf0e4a23b3744a8270d8dd48 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:44:08.455961 8178 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fe9ed18cf0e4a23b3744a8270d8dd48 [term 1 LEADER]: Becoming Leader. State: Replica: 7fe9ed18cf0e4a23b3744a8270d8dd48, State: Running, Role: LEADER
I20260928 06:44:08.456041 8173 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fe9ed18cf0e4a23b3744a8270d8dd48 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:44:08.456094 8178 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fe9ed18cf0e4a23b3744a8270d8dd48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe9ed18cf0e4a23b3744a8270d8dd48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43457 } }
I20260928 06:44:08.456488 8178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fe9ed18cf0e4a23b3744a8270d8dd48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fe9ed18cf0e4a23b3744a8270d8dd48. Latest consensus state: current_term: 1 leader_uuid: "7fe9ed18cf0e4a23b3744a8270d8dd48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe9ed18cf0e4a23b3744a8270d8dd48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43457 } } }
I20260928 06:44:08.456564 8180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fe9ed18cf0e4a23b3744a8270d8dd48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fe9ed18cf0e4a23b3744a8270d8dd48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe9ed18cf0e4a23b3744a8270d8dd48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43457 } } }
I20260928 06:44:08.456616 8178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fe9ed18cf0e4a23b3744a8270d8dd48 [sys.catalog]: This master's current role is: LEADER
I20260928 06:44:08.456655 8180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fe9ed18cf0e4a23b3744a8270d8dd48 [sys.catalog]: This master's current role is: LEADER
I20260928 06:44:08.456825 8187 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:44:08.457360 8187 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:44:08.459151 8187 catalog_manager.cc:1347] Generated new cluster ID: 15dd430f6aa0454f84cf7f4a2f376333
I20260928 06:44:08.459206 8187 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:44:08.459579 8177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:08.453910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:40993 (local address 127.0.0.1:43457)
0928 06:44:08.454090 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:08.454100 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:08.454138 (+ 38us) server_negotiation.cc:407] Connection header received
0928 06:44:08.454453 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:08.454462 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:08.454527 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:08.454674 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:08.454916 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:08.456398 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:08.457537 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:08.458063 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:08.458169 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:08.458861 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:08.458865 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:08.458865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:44:08.459033 (+ 168us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:08.459100 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:44:08.459118 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:08.459194 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:08.459334 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 06:44:08.459376 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":83,"threads_started":1}
I20260928 06:44:08.460419 19277 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:44:08.464500 8177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:08.461761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:33875 (local address 127.0.0.1:43457)
0928 06:44:08.461823 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:08.461825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:08.461935 (+ 110us) server_negotiation.cc:407] Connection header received
0928 06:44:08.461989 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:08.461990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:08.462001 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:08.462045 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:08.462189 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:08.462636 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:08.463425 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:08.463642 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:08.463694 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:08.464271 (+ 577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:08.464272 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:08.464272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:44:08.464277 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:08.464290 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:44:08.464295 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:08.464339 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:08.464413 (+ 74us) server_negotiation.cc:299] Negotiation successful
0928 06:44:08.464424 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:44:08.488941 8187 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:44:08.489466 8187 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:44:08.494894 8187 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fe9ed18cf0e4a23b3744a8270d8dd48: Generated new TSK 0
I20260928 06:44:08.495061 8187 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:44:08.501294 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 8105
W20260928 06:44:08.505403 8090 connection.cc:535] client connection to 127.0.0.1:43457 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43457): Connection reset by peer (error 104)
2026-09-28T06:44:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1233 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:44:08 dist-test-slave-b28r krb5kdc[8204](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:44:08 dist-test-slave-b28r krb5kdc[8204](info): set up 2 sockets
Sep 28 06:44:08 dist-test-slave-b28r krb5kdc[8204](info): commencing operation
krb5kdc: starting...
W20260928 06:44:09.443964 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:44:09 dist-test-slave-b28r krb5kdc[8204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:44:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:44:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 06:44:09.553618 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40291
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:44:09.632648 8217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:44:09.632850 8217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:44:09.632874 8217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:44:09.633953 8217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:44:09.633988 8217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:44:09.634001 8217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:44:09.634012 8217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:44:09.634022 8217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:44:09.635457 8217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40291
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577602522428-19277-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:44:09.635653 8217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:44:09.635815 8217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:44:09.640300 8222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:44:09.640301 8223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:44:09.640452 8217 server_base.cc:1034] running on GCE node
W20260928 06:44:09.640288 8225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:44:09.640779 8217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:44:09.641089 8217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:44:09.642239 8217 hybrid_clock.cc:648] HybridClock initialized: now 1790577849642215 us; error 37 us; skew 500 ppm
Sep 28 06:44:09 dist-test-slave-b28r krb5kdc[8204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577849, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:44:09.644776 8217 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 06:44:09.645179 8217 webserver.cc:460] Webserver started at http://127.0.0.1:37389/ using document root <none> and password file <none>
I20260928 06:44:09.645375 8217 fs_manager.cc:359] Metadata directory not provided
I20260928 06:44:09.645422 8217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:44:09.645543 8217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:44:09.646577 8217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "e5505f408b794563b07b3a9216fc28a0"
format_stamp: "Formatted at 2026-09-28 06:44:09 on dist-test-slave-b28r"
server_key: "1f7c0e6ba4bec7f01c3fedf1a7fe6fd8"
server_key_iv: "c7346a6b57c4787e20bded36e8ea79a2"
server_key_version: "encryptionkey@0"
I20260928 06:44:09.646911 8217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "e5505f408b794563b07b3a9216fc28a0"
format_stamp: "Formatted at 2026-09-28 06:44:09 on dist-test-slave-b28r"
server_key: "1f7c0e6ba4bec7f01c3fedf1a7fe6fd8"
server_key_iv: "c7346a6b57c4787e20bded36e8ea79a2"
server_key_version: "encryptionkey@0"
I20260928 06:44:09.648216 8217 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:44:09.648908 8232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:44:09.649052 8217 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:44:09.649142 8217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "e5505f408b794563b07b3a9216fc28a0"
format_stamp: "Formatted at 2026-09-28 06:44:09 on dist-test-slave-b28r"
server_key: "1f7c0e6ba4bec7f01c3fedf1a7fe6fd8"
server_key_iv: "c7346a6b57c4787e20bded36e8ea79a2"
server_key_version: "encryptionkey@0"
I20260928 06:44:09.649200 8217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:44:09.652523 8217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:44:09.654445 8217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:44:09.654577 8217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:44:09.658318 8217 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40291
I20260928 06:44:09.658349 8284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40291 every 8 connection(s)
I20260928 06:44:09.658659 8217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:44:09.659555 8285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:44:09.661787 8285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5505f408b794563b07b3a9216fc28a0: Bootstrap starting.
I20260928 06:44:09.662358 8285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5505f408b794563b07b3a9216fc28a0: Neither blocks nor log segments found. Creating new log.
I20260928 06:44:09.662617 8285 log.cc:824] T 00000000000000000000000000000000 P e5505f408b794563b07b3a9216fc28a0: Log is configured to *not* fsync() on all Append() calls
I20260928 06:44:09.663262 8285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5505f408b794563b07b3a9216fc28a0: No bootstrap required, opened a new log
I20260928 06:44:09.664844 8285 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5505f408b794563b07b3a9216fc28a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5505f408b794563b07b3a9216fc28a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40291 } }
I20260928 06:44:09.664986 8285 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5505f408b794563b07b3a9216fc28a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:44:09.665007 8285 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5505f408b794563b07b3a9216fc28a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5505f408b794563b07b3a9216fc28a0, State: Initialized, Role: FOLLOWER
I20260928 06:44:09.665154 8285 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5505f408b794563b07b3a9216fc28a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5505f408b794563b07b3a9216fc28a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40291 } }
I20260928 06:44:09.665230 8285 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5505f408b794563b07b3a9216fc28a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:44:09.665283 8285 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5505f408b794563b07b3a9216fc28a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:44:09.665328 8285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5505f408b794563b07b3a9216fc28a0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:44:09.665822 8285 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5505f408b794563b07b3a9216fc28a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5505f408b794563b07b3a9216fc28a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40291 } }
I20260928 06:44:09.665918 8285 leader_election.cc:302] T 00000000000000000000000000000000 P e5505f408b794563b07b3a9216fc28a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5505f408b794563b07b3a9216fc28a0; no voters:
I20260928 06:44:09.666064 8285 leader_election.cc:288] T 00000000000000000000000000000000 P e5505f408b794563b07b3a9216fc28a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:44:09.666126 8288 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5505f408b794563b07b3a9216fc28a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:44:09.666306 8285 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5505f408b794563b07b3a9216fc28a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:44:09.666317 8288 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5505f408b794563b07b3a9216fc28a0 [term 1 LEADER]: Becoming Leader. State: Replica: e5505f408b794563b07b3a9216fc28a0, State: Running, Role: LEADER
I20260928 06:44:09.666487 8288 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5505f408b794563b07b3a9216fc28a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5505f408b794563b07b3a9216fc28a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40291 } }
I20260928 06:44:09.666952 8290 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5505f408b794563b07b3a9216fc28a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5505f408b794563b07b3a9216fc28a0. Latest consensus state: current_term: 1 leader_uuid: "e5505f408b794563b07b3a9216fc28a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5505f408b794563b07b3a9216fc28a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40291 } } }
I20260928 06:44:09.667037 8290 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5505f408b794563b07b3a9216fc28a0 [sys.catalog]: This master's current role is: LEADER
I20260928 06:44:09.667254 8289 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5505f408b794563b07b3a9216fc28a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5505f408b794563b07b3a9216fc28a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5505f408b794563b07b3a9216fc28a0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40291 } } }
I20260928 06:44:09.667335 8289 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5505f408b794563b07b3a9216fc28a0 [sys.catalog]: This master's current role is: LEADER
I20260928 06:44:09.667472 8297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:44:09.667982 8297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:44:09.668339 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 8217
I20260928 06:44:09.668466 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:44:09.668668 19277 external_mini_cluster.cc:1428] Setting key 355624418e94edda3615c7db8dd445f2
I20260928 06:44:09.669611 8297 catalog_manager.cc:1347] Generated new cluster ID: a0e8aa9b0a48401b94c0b93dc1aa4a36
I20260928 06:44:09.669659 8297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:44:09.672341 8307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:09.669098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:40487 (local address 127.0.0.1:40291)
0928 06:44:09.669338 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:09.669348 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:09.669381 (+ 33us) server_negotiation.cc:407] Connection header received
0928 06:44:09.669494 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:09.669500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:09.671152 (+ 1652us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:09.671302 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:09.671948 (+ 646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:09.671951 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:09.671951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:44:09.671954 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:09.671988 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:44:09.672001 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:09.672055 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:09.672131 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 06:44:09.672171 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":72,"threads_started":1}
I20260928 06:44:09.673195 19277 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:44:09.675660 8307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:09.674584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.97:59953 (local address 127.0.0.1:40291)
0928 06:44:09.674616 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:09.674618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:09.674784 (+ 166us) server_negotiation.cc:407] Connection header received
0928 06:44:09.674812 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:09.674813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:09.674819 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:09.674882 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:09.675450 (+ 568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:09.675451 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:09.675451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:44:09.675456 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:09.675471 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:44:09.675476 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:09.675530 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:09.675587 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 06:44:09.675600 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:44:09.680017 8297 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:44:09.680155 8297 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:44:09.686228 8297 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5505f408b794563b07b3a9216fc28a0: Generated new TSK 0
I20260928 06:44:09.686383 8297 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:44:09.691987 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 8217
2026-09-28T06:44:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1190 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:44:09 dist-test-slave-b28r krb5kdc[8316](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:44:09 dist-test-slave-b28r krb5kdc[8316](info): set up 2 sockets
Sep 28 06:44:09 dist-test-slave-b28r krb5kdc[8316](info): commencing operation
krb5kdc: starting...
W20260928 06:44:10.644536 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:44:10 dist-test-slave-b28r krb5kdc[8316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:44:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:44:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.211.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
I20260928 06:44:10.757128 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:34565
--webserver_interface=127.18.211.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:39803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:34565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:44:10.838230 8329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:44:10.838434 8329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:44:10.838455 8329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:44:10.839627 8329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:44:10.839669 8329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:44:10.839689 8329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:44:10.839715 8329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:44:10.839730 8329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:44:10.841212 8329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:39803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:34565
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:34565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:44:10.841435 8329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:44:10.841634 8329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:44:10.846130 8337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:44:10.846120 8334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:44:10.846315 8329 server_base.cc:1034] running on GCE node
W20260928 06:44:10.846277 8335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:44:10.846577 8329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:44:10.846858 8329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:44:10.848008 8329 hybrid_clock.cc:648] HybridClock initialized: now 1790577850847983 us; error 47 us; skew 500 ppm
Sep 28 06:44:10 dist-test-slave-b28r krb5kdc[8316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577850, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:44:10.850345 8329 init.cc:377] Logged in from keytab as kudu/127.18.211.126@KRBTEST.COM (short username kudu)
I20260928 06:44:10.850788 8329 webserver.cc:460] Webserver started at http://127.18.211.126:39175/ using document root <none> and password file <none>
I20260928 06:44:10.850982 8329 fs_manager.cc:359] Metadata directory not provided
I20260928 06:44:10.851029 8329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:44:10.851171 8329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:44:10.852206 8329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "87dda96fc28c4a828dfd1edc7f99e997"
format_stamp: "Formatted at 2026-09-28 06:44:10 on dist-test-slave-b28r"
server_key: "10f7e94e25316e8e3277dc1a406b490a"
server_key_iv: "06cf8514eadcf7a67cc640148ef323b6"
server_key_version: "encryptionkey@0"
I20260928 06:44:10.852537 8329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "87dda96fc28c4a828dfd1edc7f99e997"
format_stamp: "Formatted at 2026-09-28 06:44:10 on dist-test-slave-b28r"
server_key: "10f7e94e25316e8e3277dc1a406b490a"
server_key_iv: "06cf8514eadcf7a67cc640148ef323b6"
server_key_version: "encryptionkey@0"
I20260928 06:44:10.853838 8329 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 06:44:10.854547 8344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:44:10.854758 8329 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:44:10.854847 8329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "87dda96fc28c4a828dfd1edc7f99e997"
format_stamp: "Formatted at 2026-09-28 06:44:10 on dist-test-slave-b28r"
server_key: "10f7e94e25316e8e3277dc1a406b490a"
server_key_iv: "06cf8514eadcf7a67cc640148ef323b6"
server_key_version: "encryptionkey@0"
I20260928 06:44:10.854908 8329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:44:10.867125 8329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:44:10.868877 8329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:44:10.869006 8329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:44:10.872869 8329 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:34565
I20260928 06:44:10.872901 8396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:34565 every 8 connection(s)
I20260928 06:44:10.873217 8329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:44:10.874003 8397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:44:10.876250 8397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87dda96fc28c4a828dfd1edc7f99e997: Bootstrap starting.
I20260928 06:44:10.876796 8397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87dda96fc28c4a828dfd1edc7f99e997: Neither blocks nor log segments found. Creating new log.
I20260928 06:44:10.877046 8397 log.cc:824] T 00000000000000000000000000000000 P 87dda96fc28c4a828dfd1edc7f99e997: Log is configured to *not* fsync() on all Append() calls
I20260928 06:44:10.877665 8397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87dda96fc28c4a828dfd1edc7f99e997: No bootstrap required, opened a new log
I20260928 06:44:10.879263 8397 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87dda96fc28c4a828dfd1edc7f99e997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87dda96fc28c4a828dfd1edc7f99e997" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34565 } }
I20260928 06:44:10.879398 8397 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87dda96fc28c4a828dfd1edc7f99e997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:44:10.879419 8397 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87dda96fc28c4a828dfd1edc7f99e997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87dda96fc28c4a828dfd1edc7f99e997, State: Initialized, Role: FOLLOWER
I20260928 06:44:10.879556 8397 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87dda96fc28c4a828dfd1edc7f99e997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87dda96fc28c4a828dfd1edc7f99e997" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34565 } }
I20260928 06:44:10.879614 8397 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87dda96fc28c4a828dfd1edc7f99e997 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:44:10.879634 8397 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87dda96fc28c4a828dfd1edc7f99e997 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:44:10.879679 8397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87dda96fc28c4a828dfd1edc7f99e997 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:44:10.880463 8397 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87dda96fc28c4a828dfd1edc7f99e997 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87dda96fc28c4a828dfd1edc7f99e997" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34565 } }
I20260928 06:44:10.880546 8397 leader_election.cc:302] T 00000000000000000000000000000000 P 87dda96fc28c4a828dfd1edc7f99e997 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87dda96fc28c4a828dfd1edc7f99e997; no voters:
I20260928 06:44:10.880690 8397 leader_election.cc:288] T 00000000000000000000000000000000 P 87dda96fc28c4a828dfd1edc7f99e997 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:44:10.880764 8400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87dda96fc28c4a828dfd1edc7f99e997 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:44:10.880936 8397 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87dda96fc28c4a828dfd1edc7f99e997 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:44:10.880980 8400 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87dda96fc28c4a828dfd1edc7f99e997 [term 1 LEADER]: Becoming Leader. State: Replica: 87dda96fc28c4a828dfd1edc7f99e997, State: Running, Role: LEADER
I20260928 06:44:10.881117 8400 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87dda96fc28c4a828dfd1edc7f99e997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87dda96fc28c4a828dfd1edc7f99e997" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34565 } }
I20260928 06:44:10.881527 8401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87dda96fc28c4a828dfd1edc7f99e997 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87dda96fc28c4a828dfd1edc7f99e997" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87dda96fc28c4a828dfd1edc7f99e997" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34565 } } }
I20260928 06:44:10.881642 8401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87dda96fc28c4a828dfd1edc7f99e997 [sys.catalog]: This master's current role is: LEADER
I20260928 06:44:10.881824 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 8329
I20260928 06:44:10.881523 8402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87dda96fc28c4a828dfd1edc7f99e997 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87dda96fc28c4a828dfd1edc7f99e997. Latest consensus state: current_term: 1 leader_uuid: "87dda96fc28c4a828dfd1edc7f99e997" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87dda96fc28c4a828dfd1edc7f99e997" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34565 } } }
I20260928 06:44:10.881954 8402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87dda96fc28c4a828dfd1edc7f99e997 [sys.catalog]: This master's current role is: LEADER
I20260928 06:44:10.881961 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:44:10.882257 19277 external_mini_cluster.cc:1428] Setting key 3addc3640f1b44a4185df6306a416320
W20260928 06:44:10.882756 8416 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 87dda96fc28c4a828dfd1edc7f99e997: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:44:10.882853 8416 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:44:10.882951 8409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:44:10.883157 8409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:44:10.884617 8409 catalog_manager.cc:1347] Generated new cluster ID: 273d3a837c2849189e88d6afac7bf028
I20260928 06:44:10.884683 8409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:44:10 dist-test-slave-b28r krb5kdc[8316](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577850, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:44:10.890569 8409 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:44:10.891120 8409 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:44:10.893814 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:10.882741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45673 (local address 127.18.211.126:34565)
0928 06:44:10.882936 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:10.882946 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:10.882983 (+ 37us) server_negotiation.cc:407] Connection header received
0928 06:44:10.883126 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:10.883166 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:10.883230 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:10.883383 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:10.884086 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:10.885545 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:10.886629 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:10.887262 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:10.887360 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:10.890198 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:10.890202 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:10.890203 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:10.890215 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:10.892199 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:10.892663 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:10.892666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:10.892666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:10.892768 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:10.892981 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:10.892981 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:10.892982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:10.893245 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:10.893421 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:10.893499 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 06:44:10.893595 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":83,"threads_started":1}
I20260928 06:44:10.895136 19277 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:44:10.899390 8409 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87dda96fc28c4a828dfd1edc7f99e997: Generated new TSK 0
I20260928 06:44:10.899559 8409 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:44:10.903496 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:10.896513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41815 (local address 127.18.211.126:34565)
0928 06:44:10.896552 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:10.896553 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:10.896707 (+ 154us) server_negotiation.cc:407] Connection header received
0928 06:44:10.896764 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:10.896765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:10.896776 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:10.896829 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:10.897445 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:10.898227 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:10.899205 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:10.899534 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:10.899597 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:10.900752 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:10.900754 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:10.900754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:10.900770 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:10.902317 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:10.902677 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:10.902678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:10.902678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:10.902732 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:10.902922 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:10.902923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:10.902923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:10.903039 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:10.903180 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:10.903341 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 06:44:10.903405 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:44:10.905625 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 8329
W20260928 06:44:10.909503 8314 connection.cc:535] client connection to 127.18.211.126:34565 recv error: Network error: failed to read from TLS socket (remote: 127.18.211.126:34565): Connection reset by peer (error 104)
2026-09-28T06:44:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1213 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:44:10 dist-test-slave-b28r krb5kdc[8428](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:44:10 dist-test-slave-b28r krb5kdc[8428](info): set up 2 sockets
Sep 28 06:44:10 dist-test-slave-b28r krb5kdc[8428](info): commencing operation
krb5kdc: starting...
W20260928 06:44:11.860075 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:44:11 dist-test-slave-b28r krb5kdc[8428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577851, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:44:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:44:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.211.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
I20260928 06:44:11.976150 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:42035
--webserver_interface=127.18.211.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:45023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:42035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:44:12.056232 8441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:44:12.056434 8441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:44:12.056456 8441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:44:12.057582 8441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:44:12.057621 8441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:44:12.057636 8441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:44:12.057646 8441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:44:12.057657 8441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:44:12.059130 8441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:45023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:42035
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:42035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:44:12.059352 8441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:44:12.059549 8441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:44:12.064132 8447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:44:12.064170 8441 server_base.cc:1034] running on GCE node
W20260928 06:44:12.064141 8446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:44:12.064218 8449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:44:12.064592 8441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:44:12.064865 8441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:44:12.066028 8441 hybrid_clock.cc:648] HybridClock initialized: now 1790577852066013 us; error 28 us; skew 500 ppm
Sep 28 06:44:12 dist-test-slave-b28r krb5kdc[8428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577852, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:44:12.068468 8441 init.cc:377] Logged in from keytab as kudu/127.18.211.126@KRBTEST.COM (short username kudu)
I20260928 06:44:12.068933 8441 webserver.cc:460] Webserver started at http://127.18.211.126:36505/ using document root <none> and password file <none>
I20260928 06:44:12.069123 8441 fs_manager.cc:359] Metadata directory not provided
I20260928 06:44:12.069168 8441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:44:12.069290 8441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:44:12.070318 8441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "40859cefafa14b4fad4dda8da54d1af8"
format_stamp: "Formatted at 2026-09-28 06:44:12 on dist-test-slave-b28r"
server_key: "1a1f574ef8c4b86495952d05d9eb6cf0"
server_key_iv: "575e3a6b6ad87d7d7f843dcf730d74a0"
server_key_version: "encryptionkey@0"
I20260928 06:44:12.070648 8441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "40859cefafa14b4fad4dda8da54d1af8"
format_stamp: "Formatted at 2026-09-28 06:44:12 on dist-test-slave-b28r"
server_key: "1a1f574ef8c4b86495952d05d9eb6cf0"
server_key_iv: "575e3a6b6ad87d7d7f843dcf730d74a0"
server_key_version: "encryptionkey@0"
I20260928 06:44:12.071935 8441 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260928 06:44:12.072678 8456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:44:12.072847 8441 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:44:12.072929 8441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "40859cefafa14b4fad4dda8da54d1af8"
format_stamp: "Formatted at 2026-09-28 06:44:12 on dist-test-slave-b28r"
server_key: "1a1f574ef8c4b86495952d05d9eb6cf0"
server_key_iv: "575e3a6b6ad87d7d7f843dcf730d74a0"
server_key_version: "encryptionkey@0"
I20260928 06:44:12.072988 8441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:44:12.107743 8441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:44:12.109747 8441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:44:12.109911 8441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:44:12.113896 8441 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:42035
I20260928 06:44:12.113945 8508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:42035 every 8 connection(s)
I20260928 06:44:12.114282 8441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:44:12.115192 8509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:44:12.117507 8509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40859cefafa14b4fad4dda8da54d1af8: Bootstrap starting.
I20260928 06:44:12.118074 8509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40859cefafa14b4fad4dda8da54d1af8: Neither blocks nor log segments found. Creating new log.
I20260928 06:44:12.118342 8509 log.cc:824] T 00000000000000000000000000000000 P 40859cefafa14b4fad4dda8da54d1af8: Log is configured to *not* fsync() on all Append() calls
I20260928 06:44:12.119014 8509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40859cefafa14b4fad4dda8da54d1af8: No bootstrap required, opened a new log
I20260928 06:44:12.120723 8509 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40859cefafa14b4fad4dda8da54d1af8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40859cefafa14b4fad4dda8da54d1af8" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 42035 } }
I20260928 06:44:12.120870 8509 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40859cefafa14b4fad4dda8da54d1af8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:44:12.120893 8509 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40859cefafa14b4fad4dda8da54d1af8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40859cefafa14b4fad4dda8da54d1af8, State: Initialized, Role: FOLLOWER
I20260928 06:44:12.121021 8509 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40859cefafa14b4fad4dda8da54d1af8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40859cefafa14b4fad4dda8da54d1af8" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 42035 } }
I20260928 06:44:12.121091 8509 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40859cefafa14b4fad4dda8da54d1af8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:44:12.121117 8509 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40859cefafa14b4fad4dda8da54d1af8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:44:12.121166 8509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40859cefafa14b4fad4dda8da54d1af8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:44:12.121218 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 8441
I20260928 06:44:12.121347 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:44:12.121527 19277 external_mini_cluster.cc:1428] Setting key 30357d64d2ee924ebfbf072ff3c146da
I20260928 06:44:12.122193 8509 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40859cefafa14b4fad4dda8da54d1af8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40859cefafa14b4fad4dda8da54d1af8" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 42035 } }
I20260928 06:44:12.122313 8509 leader_election.cc:302] T 00000000000000000000000000000000 P 40859cefafa14b4fad4dda8da54d1af8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40859cefafa14b4fad4dda8da54d1af8; no voters:
I20260928 06:44:12.122478 8509 leader_election.cc:288] T 00000000000000000000000000000000 P 40859cefafa14b4fad4dda8da54d1af8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:44:12.122517 8514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40859cefafa14b4fad4dda8da54d1af8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:44:12.122673 8514 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40859cefafa14b4fad4dda8da54d1af8 [term 1 LEADER]: Becoming Leader. State: Replica: 40859cefafa14b4fad4dda8da54d1af8, State: Running, Role: LEADER
I20260928 06:44:12.122759 8509 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40859cefafa14b4fad4dda8da54d1af8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:44:12.122780 8514 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40859cefafa14b4fad4dda8da54d1af8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40859cefafa14b4fad4dda8da54d1af8" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 42035 } }
I20260928 06:44:12.123126 8514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40859cefafa14b4fad4dda8da54d1af8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40859cefafa14b4fad4dda8da54d1af8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40859cefafa14b4fad4dda8da54d1af8" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 42035 } } }
I20260928 06:44:12.123271 8514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40859cefafa14b4fad4dda8da54d1af8 [sys.catalog]: This master's current role is: LEADER
I20260928 06:44:12.123436 8514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40859cefafa14b4fad4dda8da54d1af8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40859cefafa14b4fad4dda8da54d1af8. Latest consensus state: current_term: 1 leader_uuid: "40859cefafa14b4fad4dda8da54d1af8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40859cefafa14b4fad4dda8da54d1af8" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 42035 } } }
I20260928 06:44:12.123553 8514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40859cefafa14b4fad4dda8da54d1af8 [sys.catalog]: This master's current role is: LEADER
I20260928 06:44:12.123885 8522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:44:12.124394 8522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:44:12.126669 8522 catalog_manager.cc:1347] Generated new cluster ID: b6eb4e0f66284e1db4aebb1179b4a271
I20260928 06:44:12.126724 8522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:44:12.127568 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:12.122022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39123 (local address 127.18.211.126:42035)
0928 06:44:12.122212 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:12.122222 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:12.122253 (+ 31us) server_negotiation.cc:407] Connection header received
0928 06:44:12.122366 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:12.122373 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:12.122433 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:12.122571 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:12.122809 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:12.124323 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:12.125412 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:12.126041 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:12.126155 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:12.126844 (+ 689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:12.126848 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:12.126849 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:44:12.127041 (+ 192us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:12.127103 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:44:12.127118 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:12.127192 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:12.127328 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 06:44:12.127354 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":86,"threads_started":1}
I20260928 06:44:12.128587 19277 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:44:12.133550 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:12.130010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52389 (local address 127.18.211.126:42035)
0928 06:44:12.130048 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:12.130049 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:12.130174 (+ 125us) server_negotiation.cc:407] Connection header received
0928 06:44:12.130246 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:12.130247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:12.130258 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:12.130309 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:12.130439 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:12.131202 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:12.132227 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:12.132543 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:12.132597 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:12.133309 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:12.133310 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:12.133311 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:44:12.133315 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:12.133329 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:44:12.133334 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:12.133385 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:12.133460 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 06:44:12.133472 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:44:12.134976 8522 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:44:12.135711 8522 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:44:12.140741 8522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40859cefafa14b4fad4dda8da54d1af8: Generated new TSK 0
I20260928 06:44:12.140924 8522 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:44:12.144693 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 8441
W20260928 06:44:12.148810 8426 connection.cc:535] client connection to 127.18.211.126:42035 recv error: Network error: failed to read from TLS socket (remote: 127.18.211.126:42035): Connection reset by peer (error 104)
2026-09-28T06:44:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1239 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:44:12 dist-test-slave-b28r krb5kdc[8540](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:44:12 dist-test-slave-b28r krb5kdc[8540](info): set up 2 sockets
Sep 28 06:44:12 dist-test-slave-b28r krb5kdc[8540](info): commencing operation
krb5kdc: starting...
W20260928 06:44:13.091995 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:44:13 dist-test-slave-b28r krb5kdc[8540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:44:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:44:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.211.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
I20260928 06:44:13.209396 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:45295
--webserver_interface=127.18.211.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:38393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:45295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:44:13.291814 8553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:44:13.292025 8553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:44:13.292047 8553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:44:13.293177 8553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:44:13.293227 8553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:44:13.293241 8553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:44:13.293252 8553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:44:13.293262 8553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:44:13.294754 8553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:38393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:45295
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:45295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:44:13.294986 8553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:44:13.295228 8553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:44:13.299938 8561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:44:13.299976 8559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:44:13.299994 8558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:44:13.300164 8553 server_base.cc:1034] running on GCE node
I20260928 06:44:13.300485 8553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:44:13.300757 8553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:44:13.301903 8553 hybrid_clock.cc:648] HybridClock initialized: now 1790577853301886 us; error 31 us; skew 500 ppm
Sep 28 06:44:13 dist-test-slave-b28r krb5kdc[8540](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577853, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:44:13.304333 8553 init.cc:377] Logged in from keytab as kudu/127.18.211.126@KRBTEST.COM (short username kudu)
I20260928 06:44:13.304745 8553 webserver.cc:460] Webserver started at http://127.18.211.126:44553/ using document root <none> and password file <none>
I20260928 06:44:13.304949 8553 fs_manager.cc:359] Metadata directory not provided
I20260928 06:44:13.304997 8553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:44:13.305120 8553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:44:13.306134 8553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "7c1382671de5478db80153302e1501ad"
format_stamp: "Formatted at 2026-09-28 06:44:13 on dist-test-slave-b28r"
server_key: "5ed21bb164c7b335c5845a79d960a966"
server_key_iv: "288bfd4023c85be40067159523775a46"
server_key_version: "encryptionkey@0"
I20260928 06:44:13.306459 8553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "7c1382671de5478db80153302e1501ad"
format_stamp: "Formatted at 2026-09-28 06:44:13 on dist-test-slave-b28r"
server_key: "5ed21bb164c7b335c5845a79d960a966"
server_key_iv: "288bfd4023c85be40067159523775a46"
server_key_version: "encryptionkey@0"
I20260928 06:44:13.307750 8553 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:44:13.308459 8568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:44:13.308612 8553 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:44:13.308694 8553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "7c1382671de5478db80153302e1501ad"
format_stamp: "Formatted at 2026-09-28 06:44:13 on dist-test-slave-b28r"
server_key: "5ed21bb164c7b335c5845a79d960a966"
server_key_iv: "288bfd4023c85be40067159523775a46"
server_key_version: "encryptionkey@0"
I20260928 06:44:13.308760 8553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:44:13.311995 8553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:44:13.313673 8553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:44:13.313804 8553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:44:13.317557 8553 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:45295
I20260928 06:44:13.317597 8620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:45295 every 8 connection(s)
I20260928 06:44:13.317899 8553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:44:13.318809 8621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:44:13.321076 8621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c1382671de5478db80153302e1501ad: Bootstrap starting.
I20260928 06:44:13.321620 8621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c1382671de5478db80153302e1501ad: Neither blocks nor log segments found. Creating new log.
I20260928 06:44:13.321877 8621 log.cc:824] T 00000000000000000000000000000000 P 7c1382671de5478db80153302e1501ad: Log is configured to *not* fsync() on all Append() calls
I20260928 06:44:13.322638 8621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c1382671de5478db80153302e1501ad: No bootstrap required, opened a new log
I20260928 06:44:13.323963 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 8553
I20260928 06:44:13.324064 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:44:13.324226 19277 external_mini_cluster.cc:1428] Setting key 74f8319b4eed991fefae7053f34a834c
I20260928 06:44:13.324383 8621 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c1382671de5478db80153302e1501ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c1382671de5478db80153302e1501ad" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 45295 } }
I20260928 06:44:13.324563 8621 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c1382671de5478db80153302e1501ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:44:13.324636 8621 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c1382671de5478db80153302e1501ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c1382671de5478db80153302e1501ad, State: Initialized, Role: FOLLOWER
I20260928 06:44:13.324786 8621 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c1382671de5478db80153302e1501ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c1382671de5478db80153302e1501ad" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 45295 } }
I20260928 06:44:13.324910 8621 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c1382671de5478db80153302e1501ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:44:13.324959 8621 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c1382671de5478db80153302e1501ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:44:13.325002 8621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c1382671de5478db80153302e1501ad [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:44:13.326030 8621 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c1382671de5478db80153302e1501ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c1382671de5478db80153302e1501ad" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 45295 } }
I20260928 06:44:13.326145 8621 leader_election.cc:302] T 00000000000000000000000000000000 P 7c1382671de5478db80153302e1501ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c1382671de5478db80153302e1501ad; no voters:
I20260928 06:44:13.326282 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:13.324740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37973 (local address 127.18.211.126:45295)
0928 06:44:13.324922 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:13.324931 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:13.324965 (+ 34us) server_negotiation.cc:407] Connection header received
0928 06:44:13.325095 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:13.325102 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:13.325127 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:13.325238 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:13.325852 (+ 614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:13.325855 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:13.325855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:44:13.325859 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:13.325898 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:44:13.325909 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:13.325961 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:13.326040 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 06:44:13.326072 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
I20260928 06:44:13.326310 8621 leader_election.cc:288] T 00000000000000000000000000000000 P 7c1382671de5478db80153302e1501ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:44:13.326336 8626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c1382671de5478db80153302e1501ad [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:44:13.326444 8626 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c1382671de5478db80153302e1501ad [term 1 LEADER]: Becoming Leader. State: Replica: 7c1382671de5478db80153302e1501ad, State: Running, Role: LEADER
I20260928 06:44:13.326542 8621 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c1382671de5478db80153302e1501ad [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:44:13.326517 8626 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c1382671de5478db80153302e1501ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c1382671de5478db80153302e1501ad" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 45295 } }
I20260928 06:44:13.326828 8626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c1382671de5478db80153302e1501ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c1382671de5478db80153302e1501ad. Latest consensus state: current_term: 1 leader_uuid: "7c1382671de5478db80153302e1501ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c1382671de5478db80153302e1501ad" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 45295 } } }
I20260928 06:44:13.326894 8626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c1382671de5478db80153302e1501ad [sys.catalog]: This master's current role is: LEADER
I20260928 06:44:13.326889 8627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c1382671de5478db80153302e1501ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c1382671de5478db80153302e1501ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c1382671de5478db80153302e1501ad" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 45295 } } }
I20260928 06:44:13.326977 8627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c1382671de5478db80153302e1501ad [sys.catalog]: This master's current role is: LEADER
I20260928 06:44:13.327147 8631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:44:13.327442 19277 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:44:13.327708 8631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:44:13.329942 8631 catalog_manager.cc:1347] Generated new cluster ID: 50fded29bbcd42419ee2bf4dfa8e6c46
I20260928 06:44:13.329984 8631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:44:13.330557 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:13.329423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60545 (local address 127.18.211.126:45295)
0928 06:44:13.329454 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:13.329455 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:13.329596 (+ 141us) server_negotiation.cc:407] Connection header received
0928 06:44:13.329651 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:13.329652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:13.329657 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:13.329703 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:13.330360 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:13.330361 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:13.330362 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:44:13.330366 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:13.330383 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:44:13.330387 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:13.330449 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:13.330487 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 06:44:13.330500 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260928 06:44:13.340822 8631 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:44:13.340973 8631 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:44:13.347013 8631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c1382671de5478db80153302e1501ad: Generated new TSK 0
I20260928 06:44:13.347221 8631 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:44:13.348481 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 8553
2026-09-28T06:44:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1203 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:44:13 dist-test-slave-b28r krb5kdc[8652](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:44:13 dist-test-slave-b28r krb5kdc[8652](info): set up 2 sockets
Sep 28 06:44:13 dist-test-slave-b28r krb5kdc[8652](info): commencing operation
krb5kdc: starting...
W20260928 06:44:14.300024 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:44:14 dist-test-slave-b28r krb5kdc[8652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:44:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:44:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.211.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
I20260928 06:44:14.407758 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:37631
--webserver_interface=127.18.211.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:43791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:37631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:44:14.488094 8665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:44:14.488305 8665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:44:14.488327 8665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:44:14.489507 8665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:44:14.489552 8665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:44:14.489565 8665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:44:14.489576 8665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:44:14.489588 8665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:44:14.491041 8665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:43791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:37631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:37631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:44:14.491290 8665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:44:14.491495 8665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:44:14.496189 8671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:44:14.496228 8673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:44:14.496315 8665 server_base.cc:1034] running on GCE node
W20260928 06:44:14.496189 8670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:44:14.496685 8665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:44:14.496941 8665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:44:14.498088 8665 hybrid_clock.cc:648] HybridClock initialized: now 1790577854498073 us; error 41 us; skew 500 ppm
Sep 28 06:44:14 dist-test-slave-b28r krb5kdc[8652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577854, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:44:14.500491 8665 init.cc:377] Logged in from keytab as kudu/127.18.211.126@KRBTEST.COM (short username kudu)
I20260928 06:44:14.500952 8665 webserver.cc:460] Webserver started at http://127.18.211.126:41681/ using document root <none> and password file <none>
I20260928 06:44:14.501147 8665 fs_manager.cc:359] Metadata directory not provided
I20260928 06:44:14.501188 8665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:44:14.501315 8665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:44:14.502401 8665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "3ccf293bab7549319deee0501a1c3aa6"
format_stamp: "Formatted at 2026-09-28 06:44:14 on dist-test-slave-b28r"
server_key: "d75e34c6ee85e70289858e9d25169e08"
server_key_iv: "f9a570748f3edddcd34aae426492cc04"
server_key_version: "encryptionkey@0"
I20260928 06:44:14.502727 8665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "3ccf293bab7549319deee0501a1c3aa6"
format_stamp: "Formatted at 2026-09-28 06:44:14 on dist-test-slave-b28r"
server_key: "d75e34c6ee85e70289858e9d25169e08"
server_key_iv: "f9a570748f3edddcd34aae426492cc04"
server_key_version: "encryptionkey@0"
I20260928 06:44:14.504177 8665 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:44:14.505050 8680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:44:14.505182 8665 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:44:14.505261 8665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "3ccf293bab7549319deee0501a1c3aa6"
format_stamp: "Formatted at 2026-09-28 06:44:14 on dist-test-slave-b28r"
server_key: "d75e34c6ee85e70289858e9d25169e08"
server_key_iv: "f9a570748f3edddcd34aae426492cc04"
server_key_version: "encryptionkey@0"
I20260928 06:44:14.505329 8665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:44:14.527822 8665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:44:14.529750 8665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:44:14.529883 8665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:44:14.534162 8665 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:37631
I20260928 06:44:14.534540 8665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:44:14.534554 8732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:37631 every 8 connection(s)
I20260928 06:44:14.535177 8733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:44:14.537415 8733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ccf293bab7549319deee0501a1c3aa6: Bootstrap starting.
I20260928 06:44:14.537959 8733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ccf293bab7549319deee0501a1c3aa6: Neither blocks nor log segments found. Creating new log.
I20260928 06:44:14.538214 8733 log.cc:824] T 00000000000000000000000000000000 P 3ccf293bab7549319deee0501a1c3aa6: Log is configured to *not* fsync() on all Append() calls
I20260928 06:44:14.538877 8733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ccf293bab7549319deee0501a1c3aa6: No bootstrap required, opened a new log
I20260928 06:44:14.540499 8733 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ccf293bab7549319deee0501a1c3aa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ccf293bab7549319deee0501a1c3aa6" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 37631 } }
I20260928 06:44:14.540639 8733 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ccf293bab7549319deee0501a1c3aa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:44:14.540659 8733 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ccf293bab7549319deee0501a1c3aa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ccf293bab7549319deee0501a1c3aa6, State: Initialized, Role: FOLLOWER
I20260928 06:44:14.540784 8733 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ccf293bab7549319deee0501a1c3aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ccf293bab7549319deee0501a1c3aa6" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 37631 } }
I20260928 06:44:14.540851 8733 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ccf293bab7549319deee0501a1c3aa6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:44:14.540884 8733 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ccf293bab7549319deee0501a1c3aa6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:44:14.540931 8733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ccf293bab7549319deee0501a1c3aa6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:44:14.541774 8733 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ccf293bab7549319deee0501a1c3aa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ccf293bab7549319deee0501a1c3aa6" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 37631 } }
I20260928 06:44:14.541864 8733 leader_election.cc:302] T 00000000000000000000000000000000 P 3ccf293bab7549319deee0501a1c3aa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ccf293bab7549319deee0501a1c3aa6; no voters:
I20260928 06:44:14.542007 8733 leader_election.cc:288] T 00000000000000000000000000000000 P 3ccf293bab7549319deee0501a1c3aa6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:44:14.542068 8736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ccf293bab7549319deee0501a1c3aa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:44:14.542268 8733 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ccf293bab7549319deee0501a1c3aa6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:44:14.542275 8736 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ccf293bab7549319deee0501a1c3aa6 [term 1 LEADER]: Becoming Leader. State: Replica: 3ccf293bab7549319deee0501a1c3aa6, State: Running, Role: LEADER
I20260928 06:44:14.542407 8736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ccf293bab7549319deee0501a1c3aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ccf293bab7549319deee0501a1c3aa6" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 37631 } }
I20260928 06:44:14.542738 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 8665
I20260928 06:44:14.542814 8738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ccf293bab7549319deee0501a1c3aa6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ccf293bab7549319deee0501a1c3aa6. Latest consensus state: current_term: 1 leader_uuid: "3ccf293bab7549319deee0501a1c3aa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ccf293bab7549319deee0501a1c3aa6" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 37631 } } }
I20260928 06:44:14.542894 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:44:14.542889 8738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ccf293bab7549319deee0501a1c3aa6 [sys.catalog]: This master's current role is: LEADER
I20260928 06:44:14.543104 19277 external_mini_cluster.cc:1428] Setting key fd741eecc4afcd28a3afa4b70f3cb422
I20260928 06:44:14.543056 8737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ccf293bab7549319deee0501a1c3aa6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ccf293bab7549319deee0501a1c3aa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ccf293bab7549319deee0501a1c3aa6" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 37631 } } }
I20260928 06:44:14.543140 8737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ccf293bab7549319deee0501a1c3aa6 [sys.catalog]: This master's current role is: LEADER
I20260928 06:44:14.543411 8745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:44:14.543946 8745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:44:14.545436 8745 catalog_manager.cc:1347] Generated new cluster ID: 75a90de1503f4a0eba6d643542aa30eb
I20260928 06:44:14.545490 8745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 28 06:44:14 dist-test-slave-b28r krb5kdc[8652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:44:14.554984 8754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:14.543754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33517 (local address 127.18.211.126:37631)
0928 06:44:14.544535 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:14.544542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:14.544567 (+ 25us) server_negotiation.cc:407] Connection header received
0928 06:44:14.544665 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:14.544670 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:14.544713 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:14.544836 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:14.545616 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:14.546987 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:14.548083 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:14.548717 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:14.548787 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:14.551621 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:14.551625 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:14.551626 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:14.551638 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:14.553446 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:14.553842 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:14.553844 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:14.553844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:14.553943 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:14.554115 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:14.554115 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:14.554116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:14.554407 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:14.554528 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:14.554640 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 06:44:14.554768 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":59,"threads_started":1}
I20260928 06:44:14.556027 19277 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:44:14.558545 8745 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:44:14.559027 8745 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:44:14.566048 8745 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ccf293bab7549319deee0501a1c3aa6: Generated new TSK 0
I20260928 06:44:14.566217 8745 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:44:14.569847 8754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:14.558339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54759 (local address 127.18.211.126:37631)
0928 06:44:14.558372 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:14.558374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:14.558472 (+ 98us) server_negotiation.cc:407] Connection header received
0928 06:44:14.563193 (+ 4721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:14.563194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:14.563206 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:14.563232 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:14.563879 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:14.564673 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:14.565671 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:14.566011 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:14.566074 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:14.567554 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:14.567555 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:14.567556 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:14.567562 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:14.568802 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:14.569192 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:14.569193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:14.569193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:14.569253 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:14.569440 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:14.569441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:14.569441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:14.569537 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:14.569606 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:14.569717 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 06:44:14.569783 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 06:44:14.571872 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 8665
2026-09-28T06:44:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1223 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:44:14 dist-test-slave-b28r krb5kdc[8764](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:44:14 dist-test-slave-b28r krb5kdc[8764](info): set up 2 sockets
Sep 28 06:44:14 dist-test-slave-b28r krb5kdc[8764](info): commencing operation
krb5kdc: starting...
W20260928 06:44:15.515936 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:44:15 dist-test-slave-b28r krb5kdc[8764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577855, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:44:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:44:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.211.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
I20260928 06:44:15.626173 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:35569
--webserver_interface=127.18.211.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:33147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:35569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:44:15.704288 8777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:44:15.704494 8777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:44:15.704524 8777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:44:15.705650 8777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:44:15.705691 8777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:44:15.705709 8777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:44:15.705725 8777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:44:15.705739 8777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:44:15.707237 8777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:33147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:35569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:35569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:44:15.707477 8777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:44:15.707676 8777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:44:15.712119 8782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:44:15.712234 8785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:44:15.712466 8777 server_base.cc:1034] running on GCE node
W20260928 06:44:15.712481 8783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:44:15.712756 8777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:44:15.713021 8777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:44:15.714340 8777 hybrid_clock.cc:648] HybridClock initialized: now 1790577855714320 us; error 38 us; skew 500 ppm
Sep 28 06:44:15 dist-test-slave-b28r krb5kdc[8764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577855, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:44:15.716679 8777 init.cc:377] Logged in from keytab as kudu/127.18.211.126@KRBTEST.COM (short username kudu)
I20260928 06:44:15.717115 8777 webserver.cc:460] Webserver started at http://127.18.211.126:33125/ using document root <none> and password file <none>
I20260928 06:44:15.717334 8777 fs_manager.cc:359] Metadata directory not provided
I20260928 06:44:15.717382 8777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:44:15.717504 8777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:44:15.718560 8777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "10626f61ffed4879bb411461673d781c"
format_stamp: "Formatted at 2026-09-28 06:44:15 on dist-test-slave-b28r"
server_key: "2fcc7eb6b0edd6b054b93e2144f5014f"
server_key_iv: "b1c19e7e5af26ab361ac45e3d0827134"
server_key_version: "encryptionkey@0"
I20260928 06:44:15.718894 8777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "10626f61ffed4879bb411461673d781c"
format_stamp: "Formatted at 2026-09-28 06:44:15 on dist-test-slave-b28r"
server_key: "2fcc7eb6b0edd6b054b93e2144f5014f"
server_key_iv: "b1c19e7e5af26ab361ac45e3d0827134"
server_key_version: "encryptionkey@0"
I20260928 06:44:15.720275 8777 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260928 06:44:15.721004 8792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:44:15.721179 8777 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:44:15.721323 8777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "10626f61ffed4879bb411461673d781c"
format_stamp: "Formatted at 2026-09-28 06:44:15 on dist-test-slave-b28r"
server_key: "2fcc7eb6b0edd6b054b93e2144f5014f"
server_key_iv: "b1c19e7e5af26ab361ac45e3d0827134"
server_key_version: "encryptionkey@0"
I20260928 06:44:15.721393 8777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:44:15.739315 8777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:44:15.741173 8777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:44:15.741319 8777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:44:15.745260 8777 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:35569
I20260928 06:44:15.745311 8844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:35569 every 8 connection(s)
I20260928 06:44:15.745645 8777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:44:15.746462 8845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:44:15.748819 8845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10626f61ffed4879bb411461673d781c: Bootstrap starting.
I20260928 06:44:15.749378 8845 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10626f61ffed4879bb411461673d781c: Neither blocks nor log segments found. Creating new log.
I20260928 06:44:15.749627 8845 log.cc:824] T 00000000000000000000000000000000 P 10626f61ffed4879bb411461673d781c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:44:15.750341 8845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10626f61ffed4879bb411461673d781c: No bootstrap required, opened a new log
I20260928 06:44:15.750625 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 8777
I20260928 06:44:15.750738 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:44:15.750919 19277 external_mini_cluster.cc:1428] Setting key 05e6549c9ac7fc9a7e93140b6edf2b65
I20260928 06:44:15.752468 8845 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10626f61ffed4879bb411461673d781c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10626f61ffed4879bb411461673d781c" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35569 } }
I20260928 06:44:15.752643 8845 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10626f61ffed4879bb411461673d781c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:44:15.752689 8845 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10626f61ffed4879bb411461673d781c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10626f61ffed4879bb411461673d781c, State: Initialized, Role: FOLLOWER
I20260928 06:44:15.752839 8845 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10626f61ffed4879bb411461673d781c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10626f61ffed4879bb411461673d781c" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35569 } }
I20260928 06:44:15.752931 8845 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10626f61ffed4879bb411461673d781c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:44:15.752970 8845 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10626f61ffed4879bb411461673d781c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:44:15.753019 8845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10626f61ffed4879bb411461673d781c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:44:15.753746 8845 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10626f61ffed4879bb411461673d781c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10626f61ffed4879bb411461673d781c" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35569 } }
I20260928 06:44:15.753854 8845 leader_election.cc:302] T 00000000000000000000000000000000 P 10626f61ffed4879bb411461673d781c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10626f61ffed4879bb411461673d781c; no voters:
I20260928 06:44:15.753997 8845 leader_election.cc:288] T 00000000000000000000000000000000 P 10626f61ffed4879bb411461673d781c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:44:15.754030 8850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10626f61ffed4879bb411461673d781c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:44:15.754201 8850 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10626f61ffed4879bb411461673d781c [term 1 LEADER]: Becoming Leader. State: Replica: 10626f61ffed4879bb411461673d781c, State: Running, Role: LEADER
I20260928 06:44:15.754236 8845 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10626f61ffed4879bb411461673d781c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:44:15.754369 8850 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10626f61ffed4879bb411461673d781c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10626f61ffed4879bb411461673d781c" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35569 } }
I20260928 06:44:15.756316 8852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10626f61ffed4879bb411461673d781c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10626f61ffed4879bb411461673d781c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10626f61ffed4879bb411461673d781c" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35569 } } }
I20260928 06:44:15.756392 8852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10626f61ffed4879bb411461673d781c [sys.catalog]: This master's current role is: LEADER
I20260928 06:44:15.756045 8851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10626f61ffed4879bb411461673d781c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10626f61ffed4879bb411461673d781c. Latest consensus state: current_term: 1 leader_uuid: "10626f61ffed4879bb411461673d781c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10626f61ffed4879bb411461673d781c" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 35569 } } }
I20260928 06:44:15.756558 8851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10626f61ffed4879bb411461673d781c [sys.catalog]: This master's current role is: LEADER
W20260928 06:44:15.756582 8864 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 10626f61ffed4879bb411461673d781c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:44:15.756662 8864 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:44:15.757792 8849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:15.751524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53341 (local address 127.18.211.126:35569)
0928 06:44:15.751724 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:15.751735 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:15.751779 (+ 44us) server_negotiation.cc:407] Connection header received
0928 06:44:15.752434 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:15.752443 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:15.752509 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:15.752658 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:15.752915 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:15.754580 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:15.755623 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:15.756138 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:15.756243 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:15.757095 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:15.757099 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:15.757100 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:44:15.757321 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:15.757363 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:44:15.757378 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:15.757457 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:15.757521 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 06:44:15.757586 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":89,"threads_started":1}
I20260928 06:44:15.758767 19277 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:44:15.763135 8867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:44:15.763379 8867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:44:15.764887 8867 catalog_manager.cc:1347] Generated new cluster ID: c164e1ef5d954a11a9ca03b733f47761
I20260928 06:44:15.764948 8867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:44:15.766058 8849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:15.760295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33023 (local address 127.18.211.126:35569)
0928 06:44:15.760345 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:15.760347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:15.763322 (+ 2975us) server_negotiation.cc:407] Connection header received
0928 06:44:15.763343 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:15.763344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:15.763365 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:15.763404 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:15.763573 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:15.764045 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:15.765029 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:15.765226 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:15.765284 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:15.765850 (+ 566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:15.765851 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:15.765851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:44:15.765857 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:15.765868 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:44:15.765872 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:15.765917 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:15.765986 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 06:44:15.766004 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 06:44:15.775729 8867 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:44:15.776243 8867 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:44:15.779654 8867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10626f61ffed4879bb411461673d781c: Generated new TSK 0
I20260928 06:44:15.779800 8867 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:44:15.784974 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 8777
2026-09-28T06:44:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1213 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:44:15 dist-test-slave-b28r krb5kdc[8876](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:44:15 dist-test-slave-b28r krb5kdc[8876](info): set up 2 sockets
Sep 28 06:44:15 dist-test-slave-b28r krb5kdc[8876](info): commencing operation
krb5kdc: starting...
W20260928 06:44:16.732120 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:44:16 dist-test-slave-b28r krb5kdc[8876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:44:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:44:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.211.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
I20260928 06:44:16.845328 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:41329
--webserver_interface=127.18.211.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:33519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:41329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:44:16.924921 8889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:44:16.925128 8889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:44:16.925150 8889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:44:16.926276 8889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:44:16.926318 8889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:44:16.926331 8889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:44:16.926342 8889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:44:16.926354 8889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:44:16.927830 8889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:33519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:41329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:41329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:44:16.928071 8889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:44:16.928274 8889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:44:16.932951 8897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:44:16.933022 8895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:44:16.932958 8894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:44:16.933157 8889 server_base.cc:1034] running on GCE node
I20260928 06:44:16.933359 8889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:44:16.933609 8889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:44:16.934762 8889 hybrid_clock.cc:648] HybridClock initialized: now 1790577856934749 us; error 31 us; skew 500 ppm
Sep 28 06:44:16 dist-test-slave-b28r krb5kdc[8876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577856, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:44:16.937230 8889 init.cc:377] Logged in from keytab as kudu/127.18.211.126@KRBTEST.COM (short username kudu)
I20260928 06:44:16.937708 8889 webserver.cc:460] Webserver started at http://127.18.211.126:42029/ using document root <none> and password file <none>
I20260928 06:44:16.937922 8889 fs_manager.cc:359] Metadata directory not provided
I20260928 06:44:16.937974 8889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:44:16.938105 8889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:44:16.939231 8889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "d5a1e630dccf407c9bea29f27d4e6b9c"
format_stamp: "Formatted at 2026-09-28 06:44:16 on dist-test-slave-b28r"
server_key: "537d29d50a14bb2c9a3c1109082c1de9"
server_key_iv: "8db438835ab57c0ddad8ccdfc1e4fe14"
server_key_version: "encryptionkey@0"
I20260928 06:44:16.939584 8889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "d5a1e630dccf407c9bea29f27d4e6b9c"
format_stamp: "Formatted at 2026-09-28 06:44:16 on dist-test-slave-b28r"
server_key: "537d29d50a14bb2c9a3c1109082c1de9"
server_key_iv: "8db438835ab57c0ddad8ccdfc1e4fe14"
server_key_version: "encryptionkey@0"
I20260928 06:44:16.940955 8889 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:44:16.941706 8904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:44:16.941855 8889 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:44:16.941941 8889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "d5a1e630dccf407c9bea29f27d4e6b9c"
format_stamp: "Formatted at 2026-09-28 06:44:16 on dist-test-slave-b28r"
server_key: "537d29d50a14bb2c9a3c1109082c1de9"
server_key_iv: "8db438835ab57c0ddad8ccdfc1e4fe14"
server_key_version: "encryptionkey@0"
I20260928 06:44:16.942001 8889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:44:16.945295 8889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:44:16.947299 8889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:44:16.947422 8889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:44:16.951404 8889 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:41329
I20260928 06:44:16.951449 8956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:41329 every 8 connection(s)
I20260928 06:44:16.951763 8889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:44:16.952667 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:44:16.954869 8957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5a1e630dccf407c9bea29f27d4e6b9c: Bootstrap starting.
I20260928 06:44:16.955446 8957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5a1e630dccf407c9bea29f27d4e6b9c: Neither blocks nor log segments found. Creating new log.
I20260928 06:44:16.955677 8957 log.cc:824] T 00000000000000000000000000000000 P d5a1e630dccf407c9bea29f27d4e6b9c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:44:16.956290 8957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5a1e630dccf407c9bea29f27d4e6b9c: No bootstrap required, opened a new log
I20260928 06:44:16.957854 8957 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5a1e630dccf407c9bea29f27d4e6b9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a1e630dccf407c9bea29f27d4e6b9c" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41329 } }
I20260928 06:44:16.957990 8957 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5a1e630dccf407c9bea29f27d4e6b9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 06:44:16.958011 8957 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5a1e630dccf407c9bea29f27d4e6b9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5a1e630dccf407c9bea29f27d4e6b9c, State: Initialized, Role: FOLLOWER
I20260928 06:44:16.958118 8957 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5a1e630dccf407c9bea29f27d4e6b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a1e630dccf407c9bea29f27d4e6b9c" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41329 } }
I20260928 06:44:16.958194 8957 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5a1e630dccf407c9bea29f27d4e6b9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 06:44:16.958220 8957 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5a1e630dccf407c9bea29f27d4e6b9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 06:44:16.958247 8957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5a1e630dccf407c9bea29f27d4e6b9c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:44:16.959049 8957 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5a1e630dccf407c9bea29f27d4e6b9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a1e630dccf407c9bea29f27d4e6b9c" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41329 } }
I20260928 06:44:16.959197 8957 leader_election.cc:302] T 00000000000000000000000000000000 P d5a1e630dccf407c9bea29f27d4e6b9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5a1e630dccf407c9bea29f27d4e6b9c; no voters:
I20260928 06:44:16.959324 8957 leader_election.cc:288] T 00000000000000000000000000000000 P d5a1e630dccf407c9bea29f27d4e6b9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 06:44:16.959385 8960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5a1e630dccf407c9bea29f27d4e6b9c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:44:16.959544 8960 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5a1e630dccf407c9bea29f27d4e6b9c [term 1 LEADER]: Becoming Leader. State: Replica: d5a1e630dccf407c9bea29f27d4e6b9c, State: Running, Role: LEADER
I20260928 06:44:16.959578 8957 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5a1e630dccf407c9bea29f27d4e6b9c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:44:16.959697 8960 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5a1e630dccf407c9bea29f27d4e6b9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a1e630dccf407c9bea29f27d4e6b9c" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41329 } }
I20260928 06:44:16.959986 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 8889
I20260928 06:44:16.960094 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:44:16.960206 8961 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5a1e630dccf407c9bea29f27d4e6b9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5a1e630dccf407c9bea29f27d4e6b9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a1e630dccf407c9bea29f27d4e6b9c" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41329 } } }
I20260928 06:44:16.960269 8961 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5a1e630dccf407c9bea29f27d4e6b9c [sys.catalog]: This master's current role is: LEADER
I20260928 06:44:16.960276 19277 external_mini_cluster.cc:1428] Setting key 795703ff203e9106b0163b23220637c3
I20260928 06:44:16.960421 8962 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5a1e630dccf407c9bea29f27d4e6b9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5a1e630dccf407c9bea29f27d4e6b9c. Latest consensus state: current_term: 1 leader_uuid: "d5a1e630dccf407c9bea29f27d4e6b9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5a1e630dccf407c9bea29f27d4e6b9c" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41329 } } }
I20260928 06:44:16.960480 8962 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5a1e630dccf407c9bea29f27d4e6b9c [sys.catalog]: This master's current role is: LEADER
I20260928 06:44:16.960669 8969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:44:16.961222 8969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:44:16.962901 8978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:16.960984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51029 (local address 127.18.211.126:41329)
0928 06:44:16.961139 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:16.961148 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:16.961182 (+ 34us) server_negotiation.cc:407] Connection header received
0928 06:44:16.961305 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:16.961314 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:16.961347 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:16.961606 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:16.962344 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:16.962348 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:16.962348 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:44:16.962351 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:16.962387 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:44:16.962398 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:16.962493 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:16.962574 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 06:44:16.962733 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
I20260928 06:44:16.962944 8969 catalog_manager.cc:1347] Generated new cluster ID: 420e584bf658482081f8f2a72a56dc0b
I20260928 06:44:16.962976 8969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:44:16.963742 19277 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:44:16.966342 8978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:16.965099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45451 (local address 127.18.211.126:41329)
0928 06:44:16.965133 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:16.965135 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:16.965234 (+ 99us) server_negotiation.cc:407] Connection header received
0928 06:44:16.965312 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:16.965313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:16.965317 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:16.965340 (+ 23us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:16.966079 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:16.966080 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:16.966080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 06:44:16.966084 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:16.966098 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 06:44:16.966102 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:16.966133 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:16.966249 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 06:44:16.966263 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:44:16.974061 8969 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:44:16.974311 8969 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:44:16.979907 8969 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5a1e630dccf407c9bea29f27d4e6b9c: Generated new TSK 0
I20260928 06:44:16.980113 8969 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:44:16.982136 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 8889
2026-09-28T06:44:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1199 ms)
[----------] 18 tests from AuthTokenIssuingTest (21981 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:44:17 dist-test-slave-b28r krb5kdc[8988](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:44:17 dist-test-slave-b28r krb5kdc[8988](info): set up 2 sockets
Sep 28 06:44:17 dist-test-slave-b28r krb5kdc[8988](info): commencing operation
krb5kdc: starting...
W20260928 06:44:17.932199 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:44:18 dist-test-slave-b28r krb5kdc[8988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:44:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:44:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.211.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
I20260928 06:44:18.043825 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:41027
--webserver_interface=127.18.211.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:34509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:41027,127.18.211.125:39155,127.18.211.124:35321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:44:18.122929 9001 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:44:18.123171 9001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:44:18.123189 9001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:44:18.123208 9001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:44:18.124410 9001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:44:18.124456 9001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:44:18.124469 9001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:44:18.124480 9001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:44:18.124491 9001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:44:18.126011 9001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:34509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:41027,127.18.211.125:39155,127.18.211.124:35321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:41027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:44:18.126245 9001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:44:18.126466 9001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:44:18.131137 9006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:44:18.131145 9007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:44:18.131183 9009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:44:18.131520 9001 server_base.cc:1034] running on GCE node
I20260928 06:44:18.131759 9001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:44:18.132076 9001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:44:18.133271 9001 hybrid_clock.cc:648] HybridClock initialized: now 1790577858133249 us; error 38 us; skew 500 ppm
Sep 28 06:44:18 dist-test-slave-b28r krb5kdc[8988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577858, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:44:18.135835 9001 init.cc:377] Logged in from keytab as kudu/127.18.211.126@KRBTEST.COM (short username kudu)
I20260928 06:44:18.136258 9001 webserver.cc:460] Webserver started at http://127.18.211.126:42201/ using document root <none> and password file <none>
I20260928 06:44:18.136467 9001 fs_manager.cc:359] Metadata directory not provided
I20260928 06:44:18.136514 9001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:44:18.136634 9001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:44:18.137672 9001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "0669d0bb27c34da2becf35602f48d34a"
format_stamp: "Formatted at 2026-09-28 06:44:18 on dist-test-slave-b28r"
server_key: "1315bbce7985b7285ebae9bf88b7edf5"
server_key_iv: "59e4f03b14986b34aa804c9a1b311b51"
server_key_version: "encryptionkey@0"
I20260928 06:44:18.138003 9001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "0669d0bb27c34da2becf35602f48d34a"
format_stamp: "Formatted at 2026-09-28 06:44:18 on dist-test-slave-b28r"
server_key: "1315bbce7985b7285ebae9bf88b7edf5"
server_key_iv: "59e4f03b14986b34aa804c9a1b311b51"
server_key_version: "encryptionkey@0"
I20260928 06:44:18.139305 9001 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:44:18.140035 9016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:44:18.140228 9001 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 06:44:18.140309 9001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "0669d0bb27c34da2becf35602f48d34a"
format_stamp: "Formatted at 2026-09-28 06:44:18 on dist-test-slave-b28r"
server_key: "1315bbce7985b7285ebae9bf88b7edf5"
server_key_iv: "59e4f03b14986b34aa804c9a1b311b51"
server_key_version: "encryptionkey@0"
I20260928 06:44:18.140367 9001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:44:18.159554 9001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:44:18.161761 9001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:44:18.161907 9001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:44:18.166018 9001 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:41027
I20260928 06:44:18.166060 9068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:41027 every 8 connection(s)
I20260928 06:44:18.166409 9001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:44:18.167311 9069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:44:18.168345 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 9001
I20260928 06:44:18.168463 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:44:18.168663 19277 external_mini_cluster.cc:1428] Setting key 393f91e453af9d027490c395a29dc7df
I20260928 06:44:18.169214 9069 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41027 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:44:18 dist-test-slave-b28r krb5kdc[8988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577858, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.211.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.125@KRBTEST.COM" created.
I20260928 06:44:18.179332 9073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.170209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45632 (local address 127.18.211.126:41027)
0928 06:44:18.170335 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:18.170340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:18.170374 (+ 34us) server_negotiation.cc:407] Connection header received
0928 06:44:18.170866 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:18.170868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:18.170888 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:18.170967 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:18.171544 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.172350 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.173587 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.173914 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.174000 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:18.176318 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:18.176327 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:18.176328 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:18.176335 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:18.177759 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:18.178143 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:18.178146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:18.178146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:18.178222 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:18.178447 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:18.178448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:18.178448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:18.178721 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:18.178821 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:18.179019 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 06:44:18.179148 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":55,"threads_started":1}
I20260928 06:44:18.179409 9072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.169987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:41027 (local address 127.0.0.1:45632)
0928 06:44:18.170177 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:18.170226 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 06:44:18.170713 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:44:18.170976 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:44:18.170978 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:44:18.171292 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:44:18.171452 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:44:18.171457 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.172434 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.172435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:44:18.173493 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:44:18.173497 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.174002 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.174004 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:44:18.174082 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:18.174572 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:44:18.174582 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:44:18.176214 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:44:18.177858 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:44:18.177866 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:44:18.177869 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:44:18.178079 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:44:18.178269 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:44:18.178270 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:44:18.178270 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:44:18.178392 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:44:18.178800 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:44:18.178808 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:44:18.178888 (+ 80us) client_negotiation.cc:769] Sending connection context
0928 06:44:18.178994 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 06:44:18.179224 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":77,"thread_start_us":96,"threads_started":1}
I20260928 06:44:18.180364 9069 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.211.125" port: 39155 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:44:18.180694 9072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.180570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.125:39155 (local address 127.0.0.1:54258)
0928 06:44:18.180616 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:18.180670 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.125:39155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 06:44:18.180835 9020 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.211.125:39155: connect: Connection refused (error 111)
I20260928 06:44:18.181207 9072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.181136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.125:39155 (local address 127.0.0.1:54274)
0928 06:44:18.181169 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:18.181192 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.125:39155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 06:44:18.181317 9069 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.211.125:39155: Network error: Client connection negotiation failed: client connection to 127.18.211.125:39155: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.125.keytab.
Entry for principal kudu/127.18.211.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.125.keytab.
I20260928 06:44:18.192183 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.125:39155
--webserver_interface=127.18.211.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.125.keytab
--principal=kudu/127.18.211.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:34509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:41027,127.18.211.125:39155,127.18.211.124:35321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:44:18.230556 9069 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.211.125" port: 39155 } attempt: 1
I20260928 06:44:18.231047 9072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.230928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.125:39155 (local address 127.0.0.1:54278)
0928 06:44:18.230982 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:18.231016 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.125:39155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 06:44:18.231407 9072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.231325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.125:39155 (local address 127.0.0.1:54290)
0928 06:44:18.231362 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:18.231388 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.125:39155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 06:44:18.231508 9069 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.211.125:39155: Network error: Client connection negotiation failed: client connection to 127.18.211.125:39155: connect: Connection refused (error 111)
W20260928 06:44:18.273025 9076 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:44:18.273252 9076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:44:18.273275 9076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:44:18.273300 9076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:44:18.274479 9076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:44:18.274521 9076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:44:18.274540 9076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:44:18.274555 9076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:44:18.274569 9076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:44:18.276299 9076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:34509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:41027,127.18.211.125:39155,127.18.211.124:35321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.125.keytab
--principal=kudu/127.18.211.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.125:39155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.211.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:44:18.276551 9076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:44:18.276794 9076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:44:18.282747 9081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:44:18.282795 9084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:44:18.282745 9082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:44:18.283417 9076 server_base.cc:1034] running on GCE node
I20260928 06:44:18.283911 9076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:44:18.284271 9076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:44:18.285450 9076 hybrid_clock.cc:648] HybridClock initialized: now 1790577858285431 us; error 38 us; skew 500 ppm
Sep 28 06:44:18 dist-test-slave-b28r krb5kdc[8988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577858, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:44:18.287930 9076 init.cc:377] Logged in from keytab as kudu/127.18.211.125@KRBTEST.COM (short username kudu)
I20260928 06:44:18.288367 9076 webserver.cc:460] Webserver started at http://127.18.211.125:35611/ using document root <none> and password file <none>
I20260928 06:44:18.288573 9076 fs_manager.cc:359] Metadata directory not provided
I20260928 06:44:18.288625 9076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:44:18.288763 9076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:44:18.289925 9076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-1/data/instance:
uuid: "cdeb9c78015c45cb908434369142ba0c"
format_stamp: "Formatted at 2026-09-28 06:44:18 on dist-test-slave-b28r"
server_key: "920abb1e20fe8e7699d2e6d6246d0df2"
server_key_iv: "4ab41c7cedac52e097134ecc7d1284b6"
server_key_version: "encryptionkey@0"
I20260928 06:44:18.290292 9076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-1/wal/instance:
uuid: "cdeb9c78015c45cb908434369142ba0c"
format_stamp: "Formatted at 2026-09-28 06:44:18 on dist-test-slave-b28r"
server_key: "920abb1e20fe8e7699d2e6d6246d0df2"
server_key_iv: "4ab41c7cedac52e097134ecc7d1284b6"
server_key_version: "encryptionkey@0"
I20260928 06:44:18.291677 9076 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260928 06:44:18.292375 9091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:44:18.292529 9076 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:44:18.292618 9076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-1/wal
uuid: "cdeb9c78015c45cb908434369142ba0c"
format_stamp: "Formatted at 2026-09-28 06:44:18 on dist-test-slave-b28r"
server_key: "920abb1e20fe8e7699d2e6d6246d0df2"
server_key_iv: "4ab41c7cedac52e097134ecc7d1284b6"
server_key_version: "encryptionkey@0"
I20260928 06:44:18.292685 9076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:44:18.299654 9069 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.211.125" port: 39155 } attempt: 2
I20260928 06:44:18.300094 9072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.299960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.125:39155 (local address 127.0.0.1:54298)
0928 06:44:18.300023 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:18.300064 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.125:39155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 06:44:18.300390 9072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.300312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.125:39155 (local address 127.0.0.1:54314)
0928 06:44:18.300346 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:18.300371 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.125:39155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 06:44:18.300493 9069 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.211.125:39155: Network error: Client connection negotiation failed: client connection to 127.18.211.125:39155: connect: Connection refused (error 111)
I20260928 06:44:18.319041 9076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:44:18.321408 9076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:44:18.321558 9076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:44:18.325780 9076 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.125:39155
I20260928 06:44:18.325834 9143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.125:39155 every 8 connection(s)
I20260928 06:44:18.326189 9076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-1/data/info.pb
I20260928 06:44:18.326972 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 9076
I20260928 06:44:18.327075 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-1/wal/instance
I20260928 06:44:18.327137 9144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:44:18.327312 19277 external_mini_cluster.cc:1428] Setting key b82091340ad4a45cb3f8ccfc0e4727d8
I20260928 06:44:18.329136 9144 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41027 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.18.211.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.124@KRBTEST.COM" created.
Sep 28 06:44:18 dist-test-slave-b28r krb5kdc[8988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577858, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.125@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:44:18.340533 9073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.329885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45646 (local address 127.18.211.126:41027)
0928 06:44:18.329944 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:18.329946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:18.330184 (+ 238us) server_negotiation.cc:407] Connection header received
0928 06:44:18.330730 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:18.330731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:18.330746 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:18.330818 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:18.331540 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.332289 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.333359 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.333677 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.333753 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:18.337571 (+ 3818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:18.337573 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:18.337573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:18.337581 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:18.339185 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:18.339632 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:18.339633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:18.339633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:18.339681 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:18.339934 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:18.339934 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:18.339934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:18.340034 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:18.340076 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:18.340388 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 06:44:18.340466 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260928 06:44:18.340767 9147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.329875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:41027 (local address 127.0.0.1:45646)
0928 06:44:18.330053 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:18.330095 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 06:44:18.330611 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:44:18.330912 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:44:18.330917 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:44:18.331242 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:44:18.331443 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:44:18.331448 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.332387 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.332388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:44:18.333260 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:44:18.333263 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.333769 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.333770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:44:18.333886 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:18.334441 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:44:18.334450 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:44:18.337483 (+ 3033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:44:18.339270 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:44:18.339278 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:44:18.339280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:44:18.339563 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:44:18.339750 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:44:18.339751 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:44:18.339751 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:44:18.339878 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:44:18.340083 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:44:18.340089 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:44:18.340248 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 06:44:18.340359 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 06:44:18.340527 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20260928 06:44:18.341280 9144 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.211.125" port: 39155 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:44:18 dist-test-slave-b28r krb5kdc[8988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577858, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.125@KRBTEST.COM for kudu/127.18.211.125@KRBTEST.COM
I20260928 06:44:18.349416 9147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.341508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.125:39155 (local address 127.0.0.1:54316)
0928 06:44:18.341547 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:18.341557 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:44:18.341593 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:44:18.341830 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:44:18.341832 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:44:18.341969 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:44:18.342027 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:44:18.342029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.342923 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.342924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:44:18.343747 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:44:18.343750 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.344224 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.344225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:44:18.344299 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:18.344613 (+ 314us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:44:18.344618 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:44:18.346667 (+ 2049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:44:18.348494 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:44:18.348496 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:44:18.348496 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:44:18.348689 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:44:18.348887 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:44:18.348888 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:44:18.348888 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:44:18.348935 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:44:18.349190 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:44:18.349191 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:44:18.349224 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 06:44:18.349277 (+ 53us) client_negotiation.cc:240] Negotiation successful
0928 06:44:18.349353 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 06:44:18.349514 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.341577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54316 (local address 127.18.211.125:39155)
0928 06:44:18.341702 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:18.341707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:18.341723 (+ 16us) server_negotiation.cc:407] Connection header received
0928 06:44:18.341740 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:18.341742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:18.341752 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:18.341803 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:18.342096 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.342807 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.343819 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.344158 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.344209 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:18.346762 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:18.346772 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:18.346772 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:18.346779 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:18.348410 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:18.348775 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:18.348778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:18.348778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:18.348830 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:18.348970 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:18.348971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:18.348971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:18.349116 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:18.349210 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:18.349314 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 06:44:18.349400 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":101,"thread_start_us":77,"threads_started":1}
I20260928 06:44:18.349993 9144 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.211.124" port: 35321 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:44:18.350306 9147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.350204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:35321 (local address 127.0.0.1:33928)
0928 06:44:18.350245 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:18.350288 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.124:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 06:44:18.350441 9093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.211.124:35321: connect: Connection refused (error 111)
I20260928 06:44:18.350849 9147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.350773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:35321 (local address 127.0.0.1:33940)
0928 06:44:18.350810 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:18.350834 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.124:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 06:44:18.350946 9144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.211.124:35321: Network error: Client connection negotiation failed: client connection to 127.18.211.124:35321: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.124.keytab.
Entry for principal kudu/127.18.211.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.124.keytab.
I20260928 06:44:18.352052 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.124:35321
--webserver_interface=127.18.211.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.124.keytab
--principal=kudu/127.18.211.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:34509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:41027,127.18.211.125:39155,127.18.211.124:35321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:44:18.391646 9069 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.211.125" port: 39155 } attempt: 3
Sep 28 06:44:18 dist-test-slave-b28r krb5kdc[8988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577858, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for kudu/127.18.211.125@KRBTEST.COM
I20260928 06:44:18.399435 9072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.392017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.125:39155 (local address 127.0.0.1:54322)
0928 06:44:18.392072 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:18.392086 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 06:44:18.392139 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:44:18.392416 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:44:18.392419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:44:18.392588 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:44:18.392660 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:44:18.392664 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.393657 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.393658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:44:18.394222 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:44:18.394224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.394781 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.394782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:44:18.394850 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:18.395400 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:44:18.395405 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:44:18.396709 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:44:18.398423 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:44:18.398424 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:44:18.398425 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:44:18.398579 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:44:18.398788 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:44:18.398789 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:44:18.398789 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:44:18.398852 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:44:18.399164 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:44:18.399165 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:44:18.399209 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 06:44:18.399287 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 06:44:18.399363 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 06:44:18.399514 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.392107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54322 (local address 127.18.211.125:39155)
0928 06:44:18.392181 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:18.392183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:18.392195 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:44:18.392230 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:18.392231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:18.392246 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:18.392334 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:18.392755 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.393556 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.394312 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.394684 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.394759 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:18.396795 (+ 2036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:18.396797 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:18.396797 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:18.396805 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:18.398337 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:18.398640 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:18.398641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:18.398641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:18.398708 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:18.398914 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:18.398914 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:18.398915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:18.399053 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:18.399175 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:18.399329 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 06:44:18.399418 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 06:44:18.399890 9069 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.211.124" port: 35321 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:44:18.400148 9144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.211.124" port: 35321 } attempt: 1
I20260928 06:44:18.400252 9072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.400105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:35321 (local address 127.0.0.1:33956)
0928 06:44:18.400182 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:18.400222 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.124:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 06:44:18.400445 9147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.400344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:35321 (local address 127.0.0.1:33958)
0928 06:44:18.400387 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:18.400420 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.124:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260928 06:44:18.400580 9072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.400497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:35321 (local address 127.0.0.1:33974)
0928 06:44:18.400532 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:18.400554 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.124:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260928 06:44:18.400647 9069 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.211.124:35321: Network error: Client connection negotiation failed: client connection to 127.18.211.124:35321: connect: Connection refused (error 111)
I20260928 06:44:18.400763 9147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.400684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:35321 (local address 127.0.0.1:33986)
0928 06:44:18.400719 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:18.400742 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.124:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260928 06:44:18.400866 9144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.211.124:35321: Network error: Client connection negotiation failed: client connection to 127.18.211.124:35321: connect: Connection refused (error 111)
I20260928 06:44:18.431766 9069 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.211.124" port: 35321 } attempt: 1
I20260928 06:44:18.432205 9072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.432083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:35321 (local address 127.0.0.1:34000)
0928 06:44:18.432139 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:18.432177 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.124:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 06:44:18.432538 9072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.432454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:35321 (local address 127.0.0.1:34012)
0928 06:44:18.432494 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:18.432520 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.124:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 06:44:18.432643 9069 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.211.124:35321: Network error: Client connection negotiation failed: client connection to 127.18.211.124:35321: connect: Connection refused (error 111)
W20260928 06:44:18.435388 9151 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:44:18.435583 9151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:44:18.435606 9151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:44:18.435642 9151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:44:18.436842 9151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:44:18.436884 9151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:44:18.436903 9151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:44:18.436919 9151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:44:18.436934 9151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:44:18.438374 9151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:34509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:41027,127.18.211.125:39155,127.18.211.124:35321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.124.keytab
--principal=kudu/127.18.211.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.124:35321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.211.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:44:18.438603 9151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:44:18.438798 9151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:44:18.443396 9156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:44:18.443427 9159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:44:18.443574 9151 server_base.cc:1034] running on GCE node
W20260928 06:44:18.443427 9157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:44:18.444125 9151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:44:18.444398 9151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:44:18.445539 9151 hybrid_clock.cc:648] HybridClock initialized: now 1790577858445524 us; error 29 us; skew 500 ppm
Sep 28 06:44:18 dist-test-slave-b28r krb5kdc[8988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577858, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:44:18.448021 9151 init.cc:377] Logged in from keytab as kudu/127.18.211.124@KRBTEST.COM (short username kudu)
I20260928 06:44:18.448421 9151 webserver.cc:460] Webserver started at http://127.18.211.124:40659/ using document root <none> and password file <none>
I20260928 06:44:18.448611 9151 fs_manager.cc:359] Metadata directory not provided
I20260928 06:44:18.448657 9151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:44:18.448783 9151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:44:18.449807 9151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-2/data/instance:
uuid: "87c3408c226b4f50a2f28ead198e235c"
format_stamp: "Formatted at 2026-09-28 06:44:18 on dist-test-slave-b28r"
server_key: "165d39a178bfac89fb5d28f24ac0eae8"
server_key_iv: "ac055925f9ceae70844dc7715fb58954"
server_key_version: "encryptionkey@0"
I20260928 06:44:18.450127 9151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-2/wal/instance:
uuid: "87c3408c226b4f50a2f28ead198e235c"
format_stamp: "Formatted at 2026-09-28 06:44:18 on dist-test-slave-b28r"
server_key: "165d39a178bfac89fb5d28f24ac0eae8"
server_key_iv: "ac055925f9ceae70844dc7715fb58954"
server_key_version: "encryptionkey@0"
I20260928 06:44:18.451383 9151 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:44:18.452057 9166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:44:18.452196 9151 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:44:18.452277 9151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-2/wal
uuid: "87c3408c226b4f50a2f28ead198e235c"
format_stamp: "Formatted at 2026-09-28 06:44:18 on dist-test-slave-b28r"
server_key: "165d39a178bfac89fb5d28f24ac0eae8"
server_key_iv: "ac055925f9ceae70844dc7715fb58954"
server_key_version: "encryptionkey@0"
I20260928 06:44:18.452337 9151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:44:18.467291 9151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:44:18.468995 9144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.211.124" port: 35321 } attempt: 2
I20260928 06:44:18.469095 9151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:44:18.469266 9151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:44:18.469388 9147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.469272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:35321 (local address 127.0.0.1:34018)
0928 06:44:18.469320 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:18.469352 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.124:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 06:44:18.469772 9147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.469628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:35321 (local address 127.0.0.1:34032)
0928 06:44:18.469676 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:18.469711 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.124:35321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 06:44:18.469880 9144 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.211.124:35321: Network error: Client connection negotiation failed: client connection to 127.18.211.124:35321: connect: Connection refused (error 111)
I20260928 06:44:18.473153 9151 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.124:35321
I20260928 06:44:18.473196 9218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.124:35321 every 8 connection(s)
I20260928 06:44:18.473517 9151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-2/data/info.pb
I20260928 06:44:18.474344 9219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:44:18.476151 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 9151
I20260928 06:44:18.476275 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790577602522428-19277-0/minicluster-data/master-2/wal/instance
I20260928 06:44:18.476459 19277 external_mini_cluster.cc:1428] Setting key 3c77138b529586a3d17702d860eac0c2
I20260928 06:44:18.476303 9219 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41027 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:44:18 dist-test-slave-b28r krb5kdc[8988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577858, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.124@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
Sep 28 06:44:18 dist-test-slave-b28r krb5kdc[8988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.124@KRBTEST.COM
I20260928 06:44:18.485482 9073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.477041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45658 (local address 127.18.211.126:41027)
0928 06:44:18.477087 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:18.477089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:18.477245 (+ 156us) server_negotiation.cc:407] Connection header received
0928 06:44:18.477647 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:18.477648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:18.477660 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:18.477708 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:18.478262 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.478969 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.480140 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.480329 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.480385 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:18.482635 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:18.482637 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:18.482637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:18.482645 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:18.484173 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:18.484592 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:18.484593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:18.484594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:18.484653 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:18.484846 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:18.484847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:18.484847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:18.484950 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:18.485005 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:18.485278 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 06:44:18.485389 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:44:18.485595 9223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.476909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:41027 (local address 127.0.0.1:45658)
0928 06:44:18.477140 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:18.477168 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 06:44:18.477569 (+ 401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:44:18.477734 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:44:18.477736 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:44:18.477997 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:44:18.478173 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:44:18.478177 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.479052 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.479053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:44:18.480055 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:44:18.480059 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.480407 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.480408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:44:18.480517 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:18.480960 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:44:18.480969 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:44:18.482553 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:44:18.484273 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:44:18.484279 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:44:18.484281 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:44:18.484520 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:44:18.484709 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:44:18.484710 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:44:18.484710 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:44:18.484799 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:44:18.485011 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:44:18.485018 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:44:18.485149 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 06:44:18.485237 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 06:44:18.485381 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":69,"threads_started":1}
I20260928 06:44:18.486047 9219 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.211.125" port: 39155 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:44:18.486166 9222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.476873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34034 (local address 127.18.211.124:35321)
0928 06:44:18.477045 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:18.477055 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:18.477090 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:44:18.477560 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:18.477567 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:18.477587 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:18.477676 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:18.478433 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.479447 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.480425 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.480713 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.480769 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:18.483042 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:18.483050 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:18.483051 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:18.483057 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:18.484656 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:18.485100 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:18.485103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:18.485103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:18.485157 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:18.485363 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:18.485363 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:18.485364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:18.485496 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:18.485946 (+ 450us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:18.485992 (+ 46us) server_negotiation.cc:299] Negotiation successful
0928 06:44:18.486057 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":81,"threads_started":1}
Sep 28 06:44:18 dist-test-slave-b28r krb5kdc[8988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577858, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.124@KRBTEST.COM for kudu/127.18.211.125@KRBTEST.COM
Sep 28 06:44:18 dist-test-slave-b28r krb5kdc[8988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:44:18.492617 9223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.486286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.125:39155 (local address 127.0.0.1:54336)
0928 06:44:18.486370 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:18.486379 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 06:44:18.486412 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:44:18.486524 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:44:18.486524 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:44:18.486662 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:44:18.486703 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:44:18.486705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.487397 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.487398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:44:18.487930 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:44:18.487932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.488312 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.488313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:44:18.488387 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:18.488749 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:44:18.488753 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:44:18.490022 (+ 1269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:44:18.491734 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:44:18.491735 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:44:18.491735 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:44:18.491890 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:44:18.492068 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:44:18.492068 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:44:18.492068 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:44:18.492110 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:44:18.492338 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:44:18.492339 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:44:18.492382 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 06:44:18.492458 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 06:44:18.492539 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 06:44:18.492640 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.486383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54336 (local address 127.18.211.125:39155)
0928 06:44:18.486426 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:18.486428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:18.486436 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:44:18.486459 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:18.486460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:18.486469 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:18.486520 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:18.486781 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.487323 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.488014 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.488251 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.488303 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:18.490117 (+ 1814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:18.490119 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:18.490120 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:18.490128 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:18.491664 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:18.491955 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:18.491957 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:18.491957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:18.492012 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:18.492144 (+ 132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:18.492144 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:18.492145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:18.492271 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:18.492324 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:18.492479 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 06:44:18.492553 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:44:18.492990 9219 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.211.124" port: 35321 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:44:18.493372 9073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.486880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45662 (local address 127.18.211.126:41027)
0928 06:44:18.486913 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:18.486915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:18.486955 (+ 40us) server_negotiation.cc:407] Connection header received
0928 06:44:18.487008 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:18.487008 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:18.487017 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:18.487059 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:18.487523 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.487966 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.488705 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.488896 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.488946 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:18.490943 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:18.490944 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:18.490945 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:18.490951 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:18.492181 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:18.492501 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:18.492502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:18.492503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:18.492554 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:18.492761 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:18.492762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:18.492763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:18.492886 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:18.493041 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:18.493215 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 06:44:18.493288 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
Sep 28 06:44:18 dist-test-slave-b28r krb5kdc[8988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577858, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.124@KRBTEST.COM for kudu/127.18.211.124@KRBTEST.COM
Sep 28 06:44:18 dist-test-slave-b28r krb5kdc[8988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577858, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.125@KRBTEST.COM
I20260928 06:44:18.499583 9223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.493240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:35321 (local address 127.0.0.1:34042)
0928 06:44:18.493280 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:18.493288 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0928 06:44:18.493318 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:44:18.493554 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:44:18.493554 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:44:18.493669 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:44:18.493699 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:44:18.493701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.494508 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.494509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:44:18.495266 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:44:18.495268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.495703 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.495704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:44:18.495746 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:18.496084 (+ 338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:44:18.496088 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:44:18.497176 (+ 1088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:44:18.498609 (+ 1433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:44:18.498609 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:44:18.498610 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:44:18.498771 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:44:18.498955 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:44:18.498956 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:44:18.498956 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:44:18.499009 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:44:18.499276 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:44:18.499277 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:44:18.499321 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 06:44:18.499385 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 06:44:18.499469 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 06:44:18.499648 9222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.493286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34042 (local address 127.18.211.124:35321)
0928 06:44:18.493464 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:18.493466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:18.493472 (+ 6us) server_negotiation.cc:407] Connection header received
0928 06:44:18.493490 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:18.493491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:18.493498 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:18.493548 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:18.493762 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.494429 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.495331 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.495612 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.495697 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:18.497253 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:18.497254 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:18.497255 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:18.497260 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:18.498539 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:18.498840 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:18.498841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:18.498841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:18.498889 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:18.499071 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:18.499072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:18.499072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:18.499209 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:18.499265 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:18.499426 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 06:44:18.499528 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":161}
I20260928 06:44:18.500941 9219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87c3408c226b4f50a2f28ead198e235c: Bootstrap starting.
I20260928 06:44:18.501094 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.493951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54348 (local address 127.18.211.125:39155)
0928 06:44:18.493983 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:18.493984 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:18.493991 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:44:18.494035 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:18.494036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:18.494043 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:18.494082 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:18.494580 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.495312 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.496173 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.496383 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.496434 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:18.498748 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:18.498750 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:18.498750 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:18.498756 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:18.500174 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:18.500454 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:18.500456 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:18.500456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:18.500504 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:18.500658 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:18.500658 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:18.500659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:18.500757 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:18.500913 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:18.500948 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 06:44:18.501011 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260928 06:44:18.501840 9219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87c3408c226b4f50a2f28ead198e235c: Neither blocks nor log segments found. Creating new log.
I20260928 06:44:18.502096 9219 log.cc:824] T 00000000000000000000000000000000 P 87c3408c226b4f50a2f28ead198e235c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:44:18.502928 9219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87c3408c226b4f50a2f28ead198e235c: No bootstrap required, opened a new log
I20260928 06:44:18.503906 19277 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:44:18.504707 9219 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87c3408c226b4f50a2f28ead198e235c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0669d0bb27c34da2becf35602f48d34a" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41027 } } peers { permanent_uuid: "cdeb9c78015c45cb908434369142ba0c" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 39155 } } peers { permanent_uuid: "87c3408c226b4f50a2f28ead198e235c" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 35321 } }
I20260928 06:44:18.504830 9219 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87c3408c226b4f50a2f28ead198e235c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87c3408c226b4f50a2f28ead198e235c, State: Initialized, Role: FOLLOWER
I20260928 06:44:18.504905 9219 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87c3408c226b4f50a2f28ead198e235c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0669d0bb27c34da2becf35602f48d34a" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41027 } } peers { permanent_uuid: "cdeb9c78015c45cb908434369142ba0c" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 39155 } } peers { permanent_uuid: "87c3408c226b4f50a2f28ead198e235c" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 35321 } }
I20260928 06:44:18.505162 9225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87c3408c226b4f50a2f28ead198e235c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0669d0bb27c34da2becf35602f48d34a" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41027 } } peers { permanent_uuid: "cdeb9c78015c45cb908434369142ba0c" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 39155 } } peers { permanent_uuid: "87c3408c226b4f50a2f28ead198e235c" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 35321 } } }
I20260928 06:44:18.505250 9219 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87c3408c226b4f50a2f28ead198e235c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:44:18.505327 9225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87c3408c226b4f50a2f28ead198e235c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:44:18.504840 9043 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0669d0bb27c34da2becf35602f48d34a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45662
W20260928 06:44:18.506821 9236 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 87c3408c226b4f50a2f28ead198e235c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:44:18.506878 9236 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:44:18.507786 9069 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.211.124" port: 35321 } attempt: 2
I20260928 06:44:18.507056 9151 master_runner.cc:186] Error getting master registration for 127.18.211.126:41027: OK, instance_id { permanent_uuid: "0669d0bb27c34da2becf35602f48d34a" instance_seqno: 1790577858164887 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:44:18.509752 9151 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 28 06:44:18 dist-test-slave-b28r krb5kdc[8988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577858, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for kudu/127.18.211.124@KRBTEST.COM
I20260928 06:44:18.515280 9072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.508008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:35321 (local address 127.0.0.1:34056)
0928 06:44:18.508088 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:18.508098 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:44:18.508134 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:44:18.508427 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:44:18.508428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:44:18.508599 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:44:18.508647 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:44:18.508650 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.509526 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.509527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:44:18.510300 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:44:18.510303 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.510800 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.510801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:44:18.510896 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:18.511309 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:44:18.511313 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:44:18.512617 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:44:18.514223 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:44:18.514224 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:44:18.514225 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:44:18.514441 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:44:18.514633 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:44:18.514633 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:44:18.514634 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:44:18.514688 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:44:18.514964 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:44:18.514964 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:44:18.515006 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 06:44:18.515063 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 06:44:18.515178 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 06:44:18.515303 9222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.508198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34056 (local address 127.18.211.124:35321)
0928 06:44:18.508299 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:18.508301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:18.508309 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:44:18.508325 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:18.508326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:18.508334 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:18.508718 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:18.508746 (+ 28us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.509428 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.510436 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.510729 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.510793 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:18.512690 (+ 1897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:18.512692 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:18.512692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:18.512698 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:18.514144 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:18.514505 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:18.514506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:18.514506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:18.514566 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:18.514740 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:18.514741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:18.514741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:18.514898 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:18.514961 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:18.515137 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 06:44:18.515213 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 06:44:18.517417 9069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0669d0bb27c34da2becf35602f48d34a: Bootstrap starting.
I20260928 06:44:18.518116 9069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0669d0bb27c34da2becf35602f48d34a: Neither blocks nor log segments found. Creating new log.
I20260928 06:44:18.518432 9069 log.cc:824] T 00000000000000000000000000000000 P 0669d0bb27c34da2becf35602f48d34a: Log is configured to *not* fsync() on all Append() calls
I20260928 06:44:18.519289 9069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0669d0bb27c34da2becf35602f48d34a: No bootstrap required, opened a new log
I20260928 06:44:18.520622 9069 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0669d0bb27c34da2becf35602f48d34a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0669d0bb27c34da2becf35602f48d34a" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41027 } } peers { permanent_uuid: "cdeb9c78015c45cb908434369142ba0c" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 39155 } } peers { permanent_uuid: "87c3408c226b4f50a2f28ead198e235c" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 35321 } }
I20260928 06:44:18.520758 9069 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0669d0bb27c34da2becf35602f48d34a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0669d0bb27c34da2becf35602f48d34a, State: Initialized, Role: FOLLOWER
I20260928 06:44:18.520838 9069 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0669d0bb27c34da2becf35602f48d34a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0669d0bb27c34da2becf35602f48d34a" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41027 } } peers { permanent_uuid: "cdeb9c78015c45cb908434369142ba0c" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 39155 } } peers { permanent_uuid: "87c3408c226b4f50a2f28ead198e235c" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 35321 } }
I20260928 06:44:18.521143 9069 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0669d0bb27c34da2becf35602f48d34a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:44:18.521106 9239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0669d0bb27c34da2becf35602f48d34a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0669d0bb27c34da2becf35602f48d34a" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41027 } } peers { permanent_uuid: "cdeb9c78015c45cb908434369142ba0c" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 39155 } } peers { permanent_uuid: "87c3408c226b4f50a2f28ead198e235c" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 35321 } } }
I20260928 06:44:18.521189 9043 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0669d0bb27c34da2becf35602f48d34a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 06:44:18.521189 9239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0669d0bb27c34da2becf35602f48d34a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:44:18.521263 9043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0669d0bb27c34da2becf35602f48d34a [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:44:18.521885 9043 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0669d0bb27c34da2becf35602f48d34a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0669d0bb27c34da2becf35602f48d34a" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41027 } } peers { permanent_uuid: "cdeb9c78015c45cb908434369142ba0c" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 39155 } } peers { permanent_uuid: "87c3408c226b4f50a2f28ead198e235c" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 35321 } }
I20260928 06:44:18.522186 9043 leader_election.cc:288] T 00000000000000000000000000000000 P 0669d0bb27c34da2becf35602f48d34a [CANDIDATE]: Term 1 election: Requested vote from peers cdeb9c78015c45cb908434369142ba0c (127.18.211.125:39155), 87c3408c226b4f50a2f28ead198e235c (127.18.211.124:35321)
I20260928 06:44:18.522334 9193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0669d0bb27c34da2becf35602f48d34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "87c3408c226b4f50a2f28ead198e235c"
I20260928 06:44:18.522478 9193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87c3408c226b4f50a2f28ead198e235c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:44:18.523342 9193 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 87c3408c226b4f50a2f28ead198e235c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0669d0bb27c34da2becf35602f48d34a in term 1.
I20260928 06:44:18.523693 9018 leader_election.cc:302] T 00000000000000000000000000000000 P 0669d0bb27c34da2becf35602f48d34a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0669d0bb27c34da2becf35602f48d34a, 87c3408c226b4f50a2f28ead198e235c; no voters:
I20260928 06:44:18.523804 9239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0669d0bb27c34da2becf35602f48d34a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:44:18.523922 9239 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0669d0bb27c34da2becf35602f48d34a [term 1 LEADER]: Becoming Leader. State: Replica: 0669d0bb27c34da2becf35602f48d34a, State: Running, Role: LEADER
I20260928 06:44:18.524012 9239 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0669d0bb27c34da2becf35602f48d34a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0669d0bb27c34da2becf35602f48d34a" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41027 } } peers { permanent_uuid: "cdeb9c78015c45cb908434369142ba0c" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 39155 } } peers { permanent_uuid: "87c3408c226b4f50a2f28ead198e235c" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 35321 } }
I20260928 06:44:18.524412 9239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0669d0bb27c34da2becf35602f48d34a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0669d0bb27c34da2becf35602f48d34a. Latest consensus state: current_term: 1 leader_uuid: "0669d0bb27c34da2becf35602f48d34a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0669d0bb27c34da2becf35602f48d34a" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41027 } } peers { permanent_uuid: "cdeb9c78015c45cb908434369142ba0c" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 39155 } } peers { permanent_uuid: "87c3408c226b4f50a2f28ead198e235c" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 35321 } } }
I20260928 06:44:18.524487 9239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0669d0bb27c34da2becf35602f48d34a [sys.catalog]: This master's current role is: LEADER
I20260928 06:44:18.524719 9255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:44:18.525820 9255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:44:18.525877 9001 master_runner.cc:186] Error getting master registration for 127.18.211.125:39155: OK, instance_id { permanent_uuid: "cdeb9c78015c45cb908434369142ba0c" instance_seqno: 1790577858324598 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:44:18.528131 9001 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:44:18.531769 9073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.523532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45674 (local address 127.18.211.126:41027)
0928 06:44:18.523573 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:18.523575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:18.524290 (+ 715us) server_negotiation.cc:407] Connection header received
0928 06:44:18.524408 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:18.524409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:18.524418 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:18.524462 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:18.525243 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.525931 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.526944 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.527237 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.527278 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:18.528962 (+ 1684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:18.528963 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:18.528963 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:18.528970 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:18.530508 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:18.530895 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:18.530897 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:18.530897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:18.530953 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:18.531182 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:18.531183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:18.531184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:18.531307 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:18.531400 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:18.531613 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 06:44:18.531687 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260928 06:44:18.536092 9222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.523638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34068 (local address 127.18.211.124:35321)
0928 06:44:18.528610 (+ 4972us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:18.528612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:18.528622 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:44:18.528642 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:18.528643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:18.528652 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:18.529346 (+ 694us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:18.529380 (+ 34us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.530096 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.531117 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.531425 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.532743 (+ 1318us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:18.532787 (+ 44us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:18.532788 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:18.532788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:18.532793 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:18.534575 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:18.534977 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:18.534978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:18.534979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:18.535037 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:18.535281 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:18.535282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:18.535282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:18.535408 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:18.535878 (+ 470us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:18.535919 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 06:44:18.535996 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4950}
I20260928 06:44:18.538465 9193 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 87c3408c226b4f50a2f28ead198e235c [term 1 FOLLOWER]: Refusing update from remote peer 0669d0bb27c34da2becf35602f48d34a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:44:18.538800 9239 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0669d0bb27c34da2becf35602f48d34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "87c3408c226b4f50a2f28ead198e235c" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 35321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:44:18.540226 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.523928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54360 (local address 127.18.211.125:39155)
0928 06:44:18.523977 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:18.523979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:18.532650 (+ 8671us) server_negotiation.cc:407] Connection header received
0928 06:44:18.532745 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:18.532746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:18.532757 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:18.532786 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:18.533441 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.534118 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.535112 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.535422 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.535466 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:18.537080 (+ 1614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:18.537082 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:18.537082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:18.537091 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:18.538695 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:18.539318 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:18.539319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:18.539320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:18.539395 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:18.539609 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:18.539610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:18.539610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:18.539752 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:18.539825 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:18.540049 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 06:44:18.540133 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260928 06:44:18.540762 9225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87c3408c226b4f50a2f28ead198e235c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0669d0bb27c34da2becf35602f48d34a. Latest consensus state: current_term: 1 leader_uuid: "0669d0bb27c34da2becf35602f48d34a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0669d0bb27c34da2becf35602f48d34a" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41027 } } peers { permanent_uuid: "cdeb9c78015c45cb908434369142ba0c" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 39155 } } peers { permanent_uuid: "87c3408c226b4f50a2f28ead198e235c" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 35321 } } }
I20260928 06:44:18.540866 9225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87c3408c226b4f50a2f28ead198e235c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:44:18.542250 9262 mvcc.cc:205] Tried to move back new op lower bound from 7334206908570087424 to 7334206908515065856. Current Snapshot: MvccSnapshot[applied={T|T < 7334206908570087424}]
I20260928 06:44:18.542512 9239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0669d0bb27c34da2becf35602f48d34a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0669d0bb27c34da2becf35602f48d34a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0669d0bb27c34da2becf35602f48d34a" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41027 } } peers { permanent_uuid: "cdeb9c78015c45cb908434369142ba0c" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 39155 } } peers { permanent_uuid: "87c3408c226b4f50a2f28ead198e235c" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 35321 } } }
I20260928 06:44:18.542598 9239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0669d0bb27c34da2becf35602f48d34a [sys.catalog]: This master's current role is: LEADER
I20260928 06:44:18.543049 9225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87c3408c226b4f50a2f28ead198e235c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0669d0bb27c34da2becf35602f48d34a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0669d0bb27c34da2becf35602f48d34a" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41027 } } peers { permanent_uuid: "cdeb9c78015c45cb908434369142ba0c" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 39155 } } peers { permanent_uuid: "87c3408c226b4f50a2f28ead198e235c" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 35321 } } }
I20260928 06:44:18.543155 9225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87c3408c226b4f50a2f28ead198e235c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:44:18.543329 9255 catalog_manager.cc:1347] Generated new cluster ID: b1b9fe29ed64444cb7c370d5d852bddc
I20260928 06:44:18.543362 9263 mvcc.cc:205] Tried to move back new op lower bound from 7334206908570087424 to 7334206908515065856. Current Snapshot: MvccSnapshot[applied={T|T < 7334206908570087424}]
I20260928 06:44:18.543390 9255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:44:18.561002 9144 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.211.124" port: 35321 } attempt: 3
Sep 28 06:44:18 dist-test-slave-b28r krb5kdc[8988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577858, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.125@KRBTEST.COM for kudu/127.18.211.124@KRBTEST.COM
I20260928 06:44:18.571506 9255 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:44:18.572242 9255 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:44:18.573764 9147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.561310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:35321 (local address 127.0.0.1:34070)
0928 06:44:18.561434 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:18.561447 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 06:44:18.561488 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:44:18.561765 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:44:18.561767 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:44:18.561973 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:44:18.562031 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:44:18.562033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.563064 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.563066 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:44:18.563875 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:44:18.563878 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.564646 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.564647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:44:18.564730 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:18.565222 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:44:18.565241 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:44:18.566909 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:44:18.572307 (+ 5398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:44:18.572309 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:44:18.572310 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:44:18.572539 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:44:18.572935 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:44:18.572936 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:44:18.572936 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:44:18.573011 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:44:18.573436 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:44:18.573437 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:44:18.573497 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 06:44:18.573542 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 06:44:18.573653 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260928 06:44:18.574645 9222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.561555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34070 (local address 127.18.211.124:35321)
0928 06:44:18.561612 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:18.561614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:18.561620 (+ 6us) server_negotiation.cc:407] Connection header received
0928 06:44:18.561655 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:18.561656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:18.561665 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:18.562119 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:18.562173 (+ 54us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.562927 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.564026 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.564513 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.564927 (+ 414us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:18.567016 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:18.567017 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:18.567018 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:18.567024 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:18.572191 (+ 5167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:18.572721 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:18.572722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:18.572723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:18.572809 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:18.573091 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:18.573092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:18.573092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:18.573338 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:18.574262 (+ 924us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:18.574334 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 06:44:18.574470 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 06:44:18.576496 9144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdeb9c78015c45cb908434369142ba0c: Bootstrap starting.
I20260928 06:44:18.577231 9144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdeb9c78015c45cb908434369142ba0c: Neither blocks nor log segments found. Creating new log.
I20260928 06:44:18.577363 9255 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0669d0bb27c34da2becf35602f48d34a: Generated new TSK 0
I20260928 06:44:18.577523 9255 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 06:44:18.577523 9144 log.cc:824] T 00000000000000000000000000000000 P cdeb9c78015c45cb908434369142ba0c: Log is configured to *not* fsync() on all Append() calls
I20260928 06:44:18.579628 9144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdeb9c78015c45cb908434369142ba0c: No bootstrap required, opened a new log
I20260928 06:44:18.581705 9144 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdeb9c78015c45cb908434369142ba0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0669d0bb27c34da2becf35602f48d34a" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41027 } } peers { permanent_uuid: "cdeb9c78015c45cb908434369142ba0c" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 39155 } } peers { permanent_uuid: "87c3408c226b4f50a2f28ead198e235c" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 35321 } }
I20260928 06:44:18.581938 9144 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdeb9c78015c45cb908434369142ba0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdeb9c78015c45cb908434369142ba0c, State: Initialized, Role: FOLLOWER
I20260928 06:44:18.582043 9144 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdeb9c78015c45cb908434369142ba0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0669d0bb27c34da2becf35602f48d34a" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41027 } } peers { permanent_uuid: "cdeb9c78015c45cb908434369142ba0c" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 39155 } } peers { permanent_uuid: "87c3408c226b4f50a2f28ead198e235c" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 35321 } }
I20260928 06:44:18.582389 9144 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdeb9c78015c45cb908434369142ba0c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:44:18.583187 9117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdeb9c78015c45cb908434369142ba0c [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:44:18.584177 9117 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cdeb9c78015c45cb908434369142ba0c [term 1 FOLLOWER]: Refusing update from remote peer 0669d0bb27c34da2becf35602f48d34a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:44:18.584388 9118 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0669d0bb27c34da2becf35602f48d34a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cdeb9c78015c45cb908434369142ba0c"
I20260928 06:44:18.584497 9239 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0669d0bb27c34da2becf35602f48d34a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdeb9c78015c45cb908434369142ba0c" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 39155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:44:18.585176 9118 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cdeb9c78015c45cb908434369142ba0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0669d0bb27c34da2becf35602f48d34a in term 1.
I20260928 06:44:18.587373 9282 mvcc.cc:205] Tried to move back new op lower bound from 7334206908727398400 to 7334206908515065856. Current Snapshot: MvccSnapshot[applied={T|T < 7334206908570087424}]
I20260928 06:44:18.587713 9270 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdeb9c78015c45cb908434369142ba0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0669d0bb27c34da2becf35602f48d34a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0669d0bb27c34da2becf35602f48d34a" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41027 } } peers { permanent_uuid: "cdeb9c78015c45cb908434369142ba0c" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 39155 } } peers { permanent_uuid: "87c3408c226b4f50a2f28ead198e235c" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 35321 } } }
I20260928 06:44:18.587811 9270 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdeb9c78015c45cb908434369142ba0c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:44:18.587874 9270 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdeb9c78015c45cb908434369142ba0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0669d0bb27c34da2becf35602f48d34a. Latest consensus state: current_term: 1 leader_uuid: "0669d0bb27c34da2becf35602f48d34a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0669d0bb27c34da2becf35602f48d34a" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41027 } } peers { permanent_uuid: "cdeb9c78015c45cb908434369142ba0c" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 39155 } } peers { permanent_uuid: "87c3408c226b4f50a2f28ead198e235c" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 35321 } } }
I20260928 06:44:18.587941 9270 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdeb9c78015c45cb908434369142ba0c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:44:18.587972 9270 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdeb9c78015c45cb908434369142ba0c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0669d0bb27c34da2becf35602f48d34a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0669d0bb27c34da2becf35602f48d34a" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41027 } } peers { permanent_uuid: "cdeb9c78015c45cb908434369142ba0c" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 39155 } } peers { permanent_uuid: "87c3408c226b4f50a2f28ead198e235c" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 35321 } } }
I20260928 06:44:18.588012 9270 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdeb9c78015c45cb908434369142ba0c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:44:18.591712 9288 catalog_manager.cc:1259] Loaded cluster ID: b1b9fe29ed64444cb7c370d5d852bddc
I20260928 06:44:18.591765 9288 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cdeb9c78015c45cb908434369142ba0c: loading cluster ID for follower catalog manager: success
I20260928 06:44:18.592881 9288 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cdeb9c78015c45cb908434369142ba0c: acquiring CA information for follower catalog manager: success
I20260928 06:44:18.593210 9288 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cdeb9c78015c45cb908434369142ba0c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 06:44:18.600682 9073 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:44:18.599364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45684 (local address 127.18.211.126:41027)
0928 06:44:18.599408 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:18.599410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:18.600026 (+ 616us) server_negotiation.cc:407] Connection header received
0928 06:44:18.600123 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:18.600124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:18.600134 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:18.600162 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:18.600611 (+ 449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45684: BlockingRecv error: recv got EOF from 127.0.0.1:45684 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 06:44:18.601562 9149 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:44:18.599748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54376 (local address 127.18.211.125:39155)
0928 06:44:18.599787 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:18.599789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:18.601076 (+ 1287us) server_negotiation.cc:407] Connection header received
0928 06:44:18.601168 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:18.601169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:18.601179 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:18.601206 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:18.601510 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54376: BlockingRecv error: recv got EOF from 127.0.0.1:54376 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 06:44:18.602766 9222 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:44:18.602041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34086 (local address 127.18.211.124:35321)
0928 06:44:18.602080 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:18.602082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:18.602214 (+ 132us) server_negotiation.cc:407] Connection header received
0928 06:44:18.602306 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:18.602307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:18.602316 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:18.602345 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:18.602696 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34086: BlockingRecv error: recv got EOF from 127.0.0.1:34086 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:44:18.607824 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.604336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54388 (local address 127.18.211.125:39155)
0928 06:44:18.604369 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:18.604370 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:18.604598 (+ 228us) server_negotiation.cc:407] Connection header received
0928 06:44:18.604712 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:18.604713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:18.604747 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:18.604787 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:44:18.604986 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.605713 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.606826 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.607158 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.607205 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:18.607436 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:44:18.607554 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:44:18.607608 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:18.607738 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 06:44:18.607757 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 06:44:18.611157 9222 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:44:18.610334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34096 (local address 127.18.211.124:35321)
0928 06:44:18.610369 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:18.610370 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:18.610578 (+ 208us) server_negotiation.cc:407] Connection header received
0928 06:44:18.610679 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:18.610680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:18.610698 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:18.610740 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:18.611104 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34096: BlockingRecv error: recv got EOF from 127.0.0.1:34096 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:44:18.616816 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.613478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54398 (local address 127.18.211.125:39155)
0928 06:44:18.613516 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:18.613518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:18.613794 (+ 276us) server_negotiation.cc:407] Connection header received
0928 06:44:18.613889 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:18.613890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:18.613900 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:18.613927 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:44:18.614102 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.614799 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.616017 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.616303 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.616350 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:18.616549 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:44:18.616583 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:44:18.616610 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:18.616721 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 06:44:18.616744 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 06:44:18.623893 9222 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:44:18.619340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34104 (local address 127.18.211.124:35321)
0928 06:44:18.619378 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:18.619380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:18.623292 (+ 3912us) server_negotiation.cc:407] Connection header received
0928 06:44:18.623399 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:18.623400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:18.623413 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:18.623443 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:18.623832 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34104: BlockingRecv error: recv got EOF from 127.0.0.1:34104 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:44:18.630507 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.627175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54414 (local address 127.18.211.125:39155)
0928 06:44:18.627215 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:18.627217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:18.627467 (+ 250us) server_negotiation.cc:407] Connection header received
0928 06:44:18.627565 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:18.627566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:18.627575 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:18.627602 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:44:18.627805 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.628566 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.629680 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.629981 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.630028 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:18.630247 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:44:18.630283 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:44:18.630309 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:18.630411 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 06:44:18.630432 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 06:44:18.633844 9222 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:44:18.632902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34108 (local address 127.18.211.124:35321)
0928 06:44:18.632941 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:18.632942 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:18.633261 (+ 319us) server_negotiation.cc:407] Connection header received
0928 06:44:18.633356 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:18.633357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:18.633369 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:18.633396 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:18.633785 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34108: BlockingRecv error: recv got EOF from 127.0.0.1:34108 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:44:18.642107 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.639016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54416 (local address 127.18.211.125:39155)
0928 06:44:18.639055 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:18.639057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:18.639330 (+ 273us) server_negotiation.cc:407] Connection header received
0928 06:44:18.639419 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:18.639420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:18.639430 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:18.639457 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:44:18.639619 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.640292 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.641348 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.641626 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.641663 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:18.641857 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:44:18.641890 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:44:18.641914 (+ 24us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:18.642010 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 06:44:18.642028 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 06:44:18.645246 9222 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:44:18.644530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34114 (local address 127.18.211.124:35321)
0928 06:44:18.644569 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:18.644571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:18.644758 (+ 187us) server_negotiation.cc:407] Connection header received
0928 06:44:18.644845 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:18.644846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:18.644857 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:18.644885 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:18.645196 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34114: BlockingRecv error: recv got EOF from 127.0.0.1:34114 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260928 06:44:18.658047 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.654675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54422 (local address 127.18.211.125:39155)
0928 06:44:18.654729 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:18.654731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:18.655015 (+ 284us) server_negotiation.cc:407] Connection header received
0928 06:44:18.655191 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:18.655192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:18.655203 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:18.655232 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:44:18.655418 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.656160 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.657258 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.657531 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.657572 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:18.657784 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:44:18.657820 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:44:18.657845 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:18.657946 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 06:44:18.657965 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 06:44:18.661475 9222 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:44:18.660667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34122 (local address 127.18.211.124:35321)
0928 06:44:18.660737 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:18.660739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:18.660938 (+ 199us) server_negotiation.cc:407] Connection header received
0928 06:44:18.661027 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:18.661028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:18.661041 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:18.661069 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:18.661417 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34122: BlockingRecv error: recv got EOF from 127.0.0.1:34122 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:44:18.681105 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.678844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54436 (local address 127.18.211.125:39155)
0928 06:44:18.678890 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:18.678891 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:18.679012 (+ 121us) server_negotiation.cc:407] Connection header received
0928 06:44:18.679067 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:18.679067 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:18.679099 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:18.679149 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:44:18.679283 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.679771 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.680517 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.680706 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.680761 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:18.680888 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:44:18.680919 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:44:18.680957 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:18.681007 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 06:44:18.681026 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 06:44:18.686046 9222 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:44:18.685426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34124 (local address 127.18.211.124:35321)
0928 06:44:18.685468 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:18.685470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:18.685572 (+ 102us) server_negotiation.cc:407] Connection header received
0928 06:44:18.685625 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:18.685626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:18.685638 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:18.685684 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:18.685995 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34124: BlockingRecv error: recv got EOF from 127.0.0.1:34124 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:44:18.722150 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.719194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53276 (local address 127.18.211.125:39155)
0928 06:44:18.719264 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:18.719266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:18.719376 (+ 110us) server_negotiation.cc:407] Connection header received
0928 06:44:18.719456 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:18.719457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:18.719468 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:18.719528 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:44:18.719707 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.720441 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.721276 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.721640 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.721722 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:18.721882 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:44:18.721918 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:44:18.721973 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:18.722023 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 06:44:18.722042 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 06:44:18.724730 9222 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:44:18.724139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33898 (local address 127.18.211.124:35321)
0928 06:44:18.724198 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:18.724200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:18.724212 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:44:18.724318 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:18.724320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:18.724332 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:18.724378 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:18.724676 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33898: BlockingRecv error: recv got EOF from 127.0.0.1:33898 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
I20260928 06:44:18.792227 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.789961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53292 (local address 127.18.211.125:39155)
0928 06:44:18.790029 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:18.790031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:18.790167 (+ 136us) server_negotiation.cc:407] Connection header received
0928 06:44:18.790204 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:18.790206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:18.790217 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:18.790294 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:44:18.790399 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.790889 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.791702 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.791899 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.791954 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:18.792048 (+ 94us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:44:18.792076 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:44:18.792110 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:18.792145 (+ 35us) server_negotiation.cc:299] Negotiation successful
0928 06:44:18.792160 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 06:44:18.794348 9222 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:44:18.793760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33906 (local address 127.18.211.124:35321)
0928 06:44:18.793823 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:18.793825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:18.793884 (+ 59us) server_negotiation.cc:407] Connection header received
0928 06:44:18.793936 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:18.793937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:18.793947 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:18.793999 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:18.794295 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33906: BlockingRecv error: recv got EOF from 127.0.0.1:33906 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 06:44:18.926003 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:18.923527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53300 (local address 127.18.211.125:39155)
0928 06:44:18.923612 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:18.923614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:18.923630 (+ 16us) server_negotiation.cc:407] Connection header received
0928 06:44:18.923710 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:18.923711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:18.923721 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:18.923802 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:44:18.923992 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.924485 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.925390 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:18.925584 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:18.925638 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:18.925785 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:44:18.925814 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:44:18.925847 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:18.925922 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 06:44:18.925941 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 06:44:18.928339 9222 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:44:18.927740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33908 (local address 127.18.211.124:35321)
0928 06:44:18.927809 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:18.927811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:18.927824 (+ 13us) server_negotiation.cc:407] Connection header received
0928 06:44:18.927901 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:18.927902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:18.927911 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:18.927955 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:18.928274 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33908: BlockingRecv error: recv got EOF from 127.0.0.1:33908 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:44:19.003154 9239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0669d0bb27c34da2becf35602f48d34a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0669d0bb27c34da2becf35602f48d34a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0669d0bb27c34da2becf35602f48d34a" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 41027 } } peers { permanent_uuid: "cdeb9c78015c45cb908434369142ba0c" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 39155 } } peers { permanent_uuid: "87c3408c226b4f50a2f28ead198e235c" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 35321 } } }
I20260928 06:44:19.003314 9239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0669d0bb27c34da2becf35602f48d34a [sys.catalog]: This master's current role is: LEADER
I20260928 06:44:19.187996 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:19.185466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53308 (local address 127.18.211.125:39155)
0928 06:44:19.185556 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:19.185559 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:19.185647 (+ 88us) server_negotiation.cc:407] Connection header received
0928 06:44:19.185716 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:19.185717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:19.185729 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:19.185794 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:44:19.185936 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:19.186506 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:19.187341 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:19.187541 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:19.187598 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:19.187745 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:44:19.187782 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:44:19.187824 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:19.187890 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 06:44:19.187910 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 06:44:19.190508 9222 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:44:19.189970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33922 (local address 127.18.211.124:35321)
0928 06:44:19.190037 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:19.190039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:19.190054 (+ 15us) server_negotiation.cc:407] Connection header received
0928 06:44:19.190081 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:19.190081 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:19.190092 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:19.190139 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:19.190450 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33922: BlockingRecv error: recv got EOF from 127.0.0.1:33922 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 06:44:19.507352 9236 catalog_manager.cc:1259] Loaded cluster ID: b1b9fe29ed64444cb7c370d5d852bddc
I20260928 06:44:19.507441 9236 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 87c3408c226b4f50a2f28ead198e235c: loading cluster ID for follower catalog manager: success
I20260928 06:44:19.508217 9236 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 87c3408c226b4f50a2f28ead198e235c: acquiring CA information for follower catalog manager: success
I20260928 06:44:19.508468 9236 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 87c3408c226b4f50a2f28ead198e235c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:44:19.706073 9402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:19.703641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53320 (local address 127.18.211.125:39155)
0928 06:44:19.703764 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:19.703766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:19.703783 (+ 17us) server_negotiation.cc:407] Connection header received
0928 06:44:19.703849 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:19.703850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:19.703861 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:19.703925 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:44:19.704120 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:19.704651 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:19.705465 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:19.705693 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:19.705766 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:19.705880 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:44:19.705908 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:44:19.705939 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:19.705970 (+ 31us) server_negotiation.cc:299] Negotiation successful
0928 06:44:19.705986 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":77,"threads_started":1}
I20260928 06:44:19.709857 9408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:19.707632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33930 (local address 127.18.211.124:35321)
0928 06:44:19.707725 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:19.707727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:19.707735 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:44:19.707759 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:19.707759 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:19.707808 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:19.707867 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:44:19.707945 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:19.708465 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:19.709123 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:19.709361 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:19.709415 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:19.709530 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:44:19.709640 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:44:19.709691 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:19.709781 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 06:44:19.709796 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72,"thread_start_us":50,"threads_started":1}
I20260928 06:44:19.714141 9402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:19.711508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53322 (local address 127.18.211.125:39155)
0928 06:44:19.711545 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:19.711547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:19.711634 (+ 87us) server_negotiation.cc:407] Connection header received
0928 06:44:19.711712 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:19.711714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:19.711723 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:19.711780 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:44:19.711961 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:19.712415 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:19.713382 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:19.713666 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:19.713717 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:19.713886 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:44:19.713920 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:44:19.713963 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:19.714047 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 06:44:19.714071 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:44:19.714184 9415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:19.711399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43866 (local address 127.18.211.126:41027)
0928 06:44:19.711509 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:19.711511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:19.711519 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:44:19.711575 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:19.711575 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:19.711632 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:19.711718 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:44:19.711900 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:19.712649 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:19.713341 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:19.713636 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:19.713696 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:19.713870 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:44:19.713973 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:44:19.714029 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:19.714083 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 06:44:19.714104 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":61,"threads_started":1}
I20260928 06:44:19.714212 9408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:19.711387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33934 (local address 127.18.211.124:35321)
0928 06:44:19.711426 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:19.711428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:19.711630 (+ 202us) server_negotiation.cc:407] Connection header received
0928 06:44:19.711661 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:19.711662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:19.711672 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:19.711717 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:44:19.711879 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:19.712585 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:19.713528 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:19.713759 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:19.713817 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:19.713968 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:44:19.713998 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:44:19.714044 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:19.714135 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 06:44:19.714156 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:44:19.720329 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 9001
I20260928 06:44:19.725440 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 9076
I20260928 06:44:19.730094 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 9151
2026-09-28T06:44:19Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2747 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 06:44:19 dist-test-slave-b28r krb5kdc[9419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 06:44:19 dist-test-slave-b28r krb5kdc[9419](info): set up 2 sockets
Sep 28 06:44:19 dist-test-slave-b28r krb5kdc[9419](info): commencing operation
krb5kdc: starting...
W20260928 06:44:20.688148 19277 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/krb5kdc/test-user.keytab.
Sep 28 06:44:20 dist-test-slave-b28r krb5kdc[9419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T06:44:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T06:44:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.211.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
Entry for principal kudu/127.18.211.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab.
I20260928 06:44:20.795851 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:34659
--webserver_interface=127.18.211.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:41339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:34659,127.18.211.125:37627,127.18.211.124:46023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 06:44:20.874583 9432 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:44:20.874792 9432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:44:20.874815 9432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:44:20.874840 9432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:44:20.876014 9432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:44:20.876055 9432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:44:20.876072 9432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:44:20.876091 9432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:44:20.876104 9432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:44:20.877568 9432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:41339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:34659,127.18.211.125:37627,127.18.211.124:46023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.126.keytab
--principal=kudu/127.18.211.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.126:34659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.211.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:44:20.877811 9432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:44:20.878011 9432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:44:20.882557 9438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:44:20.882552 9440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:44:20.882632 9437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:44:20.882725 9432 server_base.cc:1034] running on GCE node
I20260928 06:44:20.883072 9432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:44:20.883387 9432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:44:20.884532 9432 hybrid_clock.cc:648] HybridClock initialized: now 1790577860884512 us; error 34 us; skew 500 ppm
Sep 28 06:44:20 dist-test-slave-b28r krb5kdc[9419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577860, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:44:20.886767 9432 init.cc:377] Logged in from keytab as kudu/127.18.211.126@KRBTEST.COM (short username kudu)
I20260928 06:44:20.887250 9432 webserver.cc:460] Webserver started at http://127.18.211.126:40881/ using document root <none> and password file <none>
I20260928 06:44:20.887434 9432 fs_manager.cc:359] Metadata directory not provided
I20260928 06:44:20.887473 9432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:44:20.887560 9432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:44:20.888540 9432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-0/data/instance:
uuid: "446547d6cc9142409a2a7fed33e2d931"
format_stamp: "Formatted at 2026-09-28 06:44:20 on dist-test-slave-b28r"
server_key: "288ca63a21e8b9d52c0ae8143f305827"
server_key_iv: "93d70837bec45fc8e3950e50aa4d7ea4"
server_key_version: "encryptionkey@0"
I20260928 06:44:20.888831 9432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-0/wal/instance:
uuid: "446547d6cc9142409a2a7fed33e2d931"
format_stamp: "Formatted at 2026-09-28 06:44:20 on dist-test-slave-b28r"
server_key: "288ca63a21e8b9d52c0ae8143f305827"
server_key_iv: "93d70837bec45fc8e3950e50aa4d7ea4"
server_key_version: "encryptionkey@0"
I20260928 06:44:20.890050 9432 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:44:20.890738 9447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:44:20.890928 9432 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260928 06:44:20.891012 9432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-0/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-0/wal
uuid: "446547d6cc9142409a2a7fed33e2d931"
format_stamp: "Formatted at 2026-09-28 06:44:20 on dist-test-slave-b28r"
server_key: "288ca63a21e8b9d52c0ae8143f305827"
server_key_iv: "93d70837bec45fc8e3950e50aa4d7ea4"
server_key_version: "encryptionkey@0"
I20260928 06:44:20.891065 9432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:44:20.908521 9432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:44:20.910492 9432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:44:20.910604 9432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:44:20.914381 9432 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.126:34659
I20260928 06:44:20.914433 9499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.126:34659 every 8 connection(s)
I20260928 06:44:20.914742 9432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-0/data/info.pb
I20260928 06:44:20.915619 9500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:44:20.917186 9500 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34659 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:44:20.920035 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 9432
I20260928 06:44:20.920123 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-0/wal/instance
I20260928 06:44:20.920315 19277 external_mini_cluster.cc:1428] Setting key 02a68c100bc293ff0620c23e151a720d
Sep 28 06:44:20 dist-test-slave-b28r krb5kdc[9419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577860, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:44:20.925976 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:20.917829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34659 (local address 127.0.0.1:33752)
0928 06:44:20.917980 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:20.918016 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 06:44:20.918473 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:44:20.918699 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:44:20.918702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:44:20.918933 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:44:20.919116 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:44:20.919122 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:44:20.920096 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:44:20.920097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:44:20.920866 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:44:20.920869 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:44:20.921307 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:44:20.921308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:44:20.921375 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:20.921826 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:44:20.921834 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:44:20.923146 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:44:20.924660 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:44:20.924667 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:44:20.924669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:44:20.924866 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:44:20.925070 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:44:20.925071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:44:20.925071 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:44:20.925147 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:44:20.925448 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:44:20.925453 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:44:20.925513 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 06:44:20.925592 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 06:44:20.925802 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":82,"thread_start_us":66,"threads_started":1}
I20260928 06:44:20.925979 9503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:20.917858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33752 (local address 127.18.211.126:34659)
0928 06:44:20.918019 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:20.918027 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:20.918096 (+ 69us) server_negotiation.cc:407] Connection header received
0928 06:44:20.918591 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:20.918593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:20.918609 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:20.918684 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:20.919197 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:20.920011 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:20.920946 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:20.921234 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:20.921326 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:20.923224 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:20.923231 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:20.923232 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:20.923237 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:20.924582 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:20.924939 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:20.924943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:20.924943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:20.925014 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:20.925198 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:20.925199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:20.925199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:20.925389 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:20.925457 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:20.925632 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 06:44:20.925732 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
I20260928 06:44:20.926635 9500 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37627 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.18.211.125@KRBTEST.COM; defaulting to no policy
I20260928 06:44:20.926954 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:20.926848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.125:37627 (local address 127.0.0.1:46888)
0928 06:44:20.926883 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:20.926932 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.125:37627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260928 06:44:20.927107 9451 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.211.125:37627: connect: Connection refused (error 111)
I20260928 06:44:20.927534 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:20.927451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.125:37627 (local address 127.0.0.1:46894)
0928 06:44:20.927494 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:20.927518 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.125:37627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 06:44:20.927647 9500 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.211.125:37627: Network error: Client connection negotiation failed: client connection to 127.18.211.125:37627: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.125.keytab.
Entry for principal kudu/127.18.211.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.125.keytab.
I20260928 06:44:20.942065 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.125:37627
--webserver_interface=127.18.211.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.125.keytab
--principal=kudu/127.18.211.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:41339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:34659,127.18.211.125:37627,127.18.211.124:46023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:44:20.976915 9500 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37627 } attempt: 1
I20260928 06:44:20.977434 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:20.977308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.125:37627 (local address 127.0.0.1:46904)
0928 06:44:20.977368 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:20.977403 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.125:37627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 06:44:20.977921 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:20.977826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.125:37627 (local address 127.0.0.1:46918)
0928 06:44:20.977873 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:20.977902 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.125:37627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 06:44:20.978016 9500 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.211.125:37627: Network error: Client connection negotiation failed: client connection to 127.18.211.125:37627: connect: Connection refused (error 111)
W20260928 06:44:21.022691 9507 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:44:21.022903 9507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:44:21.022925 9507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:44:21.022949 9507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:44:21.024114 9507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:44:21.024154 9507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:44:21.024173 9507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:44:21.024188 9507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:44:21.024204 9507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:44:21.025628 9507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:41339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:34659,127.18.211.125:37627,127.18.211.124:46023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.125.keytab
--principal=kudu/127.18.211.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.125:37627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.211.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:44:21.025880 9507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:44:21.026090 9507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 06:44:21.030524 9515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:44:21.030534 9513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:44:21.030534 9512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:44:21.030870 9507 server_base.cc:1034] running on GCE node
I20260928 06:44:21.031134 9507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:44:21.031438 9507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:44:21.032603 9507 hybrid_clock.cc:648] HybridClock initialized: now 1790577861032590 us; error 29 us; skew 500 ppm
Sep 28 06:44:21 dist-test-slave-b28r krb5kdc[9419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577861, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:44:21.035037 9507 init.cc:377] Logged in from keytab as kudu/127.18.211.125@KRBTEST.COM (short username kudu)
I20260928 06:44:21.035463 9507 webserver.cc:460] Webserver started at http://127.18.211.125:45857/ using document root <none> and password file <none>
I20260928 06:44:21.035662 9507 fs_manager.cc:359] Metadata directory not provided
I20260928 06:44:21.035707 9507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:44:21.035828 9507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:44:21.036881 9507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-1/data/instance:
uuid: "b69ba13be608409d84d54f0b75970732"
format_stamp: "Formatted at 2026-09-28 06:44:21 on dist-test-slave-b28r"
server_key: "6e844387a29897baa8ee91b35f2b73a4"
server_key_iv: "c00cd91cfdace15bf400a79a57271f88"
server_key_version: "encryptionkey@0"
I20260928 06:44:21.037207 9507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-1/wal/instance:
uuid: "b69ba13be608409d84d54f0b75970732"
format_stamp: "Formatted at 2026-09-28 06:44:21 on dist-test-slave-b28r"
server_key: "6e844387a29897baa8ee91b35f2b73a4"
server_key_iv: "c00cd91cfdace15bf400a79a57271f88"
server_key_version: "encryptionkey@0"
I20260928 06:44:21.038554 9507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260928 06:44:21.039397 9522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:44:21.039568 9507 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 06:44:21.039651 9507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-1/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-1/wal
uuid: "b69ba13be608409d84d54f0b75970732"
format_stamp: "Formatted at 2026-09-28 06:44:21 on dist-test-slave-b28r"
server_key: "6e844387a29897baa8ee91b35f2b73a4"
server_key_iv: "c00cd91cfdace15bf400a79a57271f88"
server_key_version: "encryptionkey@0"
I20260928 06:44:21.039721 9507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:44:21.046161 9500 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37627 } attempt: 2
I20260928 06:44:21.046622 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:21.046493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.125:37627 (local address 127.0.0.1:46920)
0928 06:44:21.046549 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:21.046591 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.125:37627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260928 06:44:21.046934 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:21.046861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.125:37627 (local address 127.0.0.1:46928)
0928 06:44:21.046897 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:21.046918 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.125:37627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 06:44:21.047065 9500 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.211.125:37627: Network error: Client connection negotiation failed: client connection to 127.18.211.125:37627: connect: Connection refused (error 111)
I20260928 06:44:21.056689 9507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:44:21.058713 9507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:44:21.058856 9507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:44:21.062705 9507 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.125:37627
I20260928 06:44:21.062749 9574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.125:37627 every 8 connection(s)
I20260928 06:44:21.063136 9507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-1/data/info.pb
I20260928 06:44:21.064026 9575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:44:21.065618 9575 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34659 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:44:21.066596 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 9507
I20260928 06:44:21.066675 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-1/wal/instance
I20260928 06:44:21.066830 19277 external_mini_cluster.cc:1428] Setting key 44ae69ad88b2bd9082c4bb997501598e
Sep 28 06:44:21 dist-test-slave-b28r krb5kdc[9419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577861, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.125@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.211.124@KRBTEST.COM; defaulting to no policy
I20260928 06:44:21.075393 9503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:21.066401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33762 (local address 127.18.211.126:34659)
0928 06:44:21.066489 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:21.066491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:21.067345 (+ 854us) server_negotiation.cc:407] Connection header received
0928 06:44:21.067979 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:21.067980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:21.067994 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:21.068050 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:21.068779 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.069252 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.070283 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.070480 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.070536 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:21.072567 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:21.072569 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:21.072569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:21.072577 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:21.074035 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:21.074413 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:21.074415 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:21.074415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:21.074472 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:21.074746 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:21.074747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:21.074748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:21.074873 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:21.074935 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:21.075238 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 06:44:21.075313 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 06:44:21.075510 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:21.066286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34659 (local address 127.0.0.1:33762)
0928 06:44:21.067170 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:21.067225 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 06:44:21.067864 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:44:21.068133 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:44:21.068138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:44:21.068478 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:44:21.068680 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:44:21.068685 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.069344 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.069345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:44:21.070195 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:44:21.070198 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.070551 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.070552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:44:21.070674 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:21.071170 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:44:21.071185 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:44:21.072481 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:44:21.074103 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:44:21.074112 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:44:21.074114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:44:21.074334 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:44:21.074526 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:44:21.074527 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:44:21.074528 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:44:21.074644 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:44:21.074933 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:44:21.074937 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:44:21.075059 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 06:44:21.075184 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 06:44:21.075308 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.211.124@KRBTEST.COM" created.
I20260928 06:44:21.076097 9575 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37627 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:44:21 dist-test-slave-b28r krb5kdc[9419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577861, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.125@KRBTEST.COM for kudu/127.18.211.125@KRBTEST.COM
I20260928 06:44:21.083431 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:21.076352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.125:37627 (local address 127.0.0.1:46944)
0928 06:44:21.076395 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:21.076406 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:44:21.076447 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:44:21.076728 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:44:21.076729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:44:21.076837 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:44:21.076886 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:44:21.076887 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.077914 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.077915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:44:21.078632 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:44:21.078635 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.079114 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.079115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:44:21.079192 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:21.079558 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:44:21.079563 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:44:21.080846 (+ 1283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:44:21.082343 (+ 1497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:44:21.082344 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:44:21.082344 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:44:21.082539 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:44:21.082719 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:44:21.082720 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:44:21.082721 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:44:21.082780 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:44:21.083035 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:44:21.083036 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:44:21.083108 (+ 72us) client_negotiation.cc:769] Sending connection context
0928 06:44:21.083185 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 06:44:21.083284 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 06:44:21.083447 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:21.076422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46944 (local address 127.18.211.125:37627)
0928 06:44:21.076570 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:21.076574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:21.076597 (+ 23us) server_negotiation.cc:407] Connection header received
0928 06:44:21.076621 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:21.076623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:21.076637 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:21.076721 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:21.076946 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.077825 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.078716 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.079020 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.079067 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:21.080926 (+ 1859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:21.080933 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:21.080933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:21.080939 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:21.082267 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:21.082608 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:21.082611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:21.082611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:21.082657 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:21.082839 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:21.082839 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:21.082840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:21.082978 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:21.083034 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:21.083209 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 06:44:21.083330 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
I20260928 06:44:21.084095 9575 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.211.124" port: 46023 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:44:21.084465 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:21.084374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:46023 (local address 127.0.0.1:46076)
0928 06:44:21.084404 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:21.084445 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.124:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260928 06:44:21.084600 9523 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.211.124:46023: connect: Connection refused (error 111)
I20260928 06:44:21.084969 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:21.084901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:46023 (local address 127.0.0.1:46078)
0928 06:44:21.084930 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:21.084954 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.124:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260928 06:44:21.085084 9575 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.211.124:46023: Network error: Client connection negotiation failed: client connection to 127.18.211.124:46023: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.211.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.124.keytab.
Entry for principal kudu/127.18.211.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.124.keytab.
I20260928 06:44:21.089769 19277 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskOUYCUS/build/release/bin/kudu
/tmp/dist-test-taskOUYCUS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.124:46023
--webserver_interface=127.18.211.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.124.keytab
--principal=kudu/127.18.211.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.211.84:41339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.211.126:34659,127.18.211.125:37627,127.18.211.124:46023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 06:44:21.134336 9575 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.211.124" port: 46023 } attempt: 1
I20260928 06:44:21.134922 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:21.134747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:46023 (local address 127.0.0.1:46086)
0928 06:44:21.134822 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:21.134878 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.124:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 06:44:21.135483 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:21.135380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:46023 (local address 127.0.0.1:46100)
0928 06:44:21.135426 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:21.135459 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.124:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260928 06:44:21.135623 9575 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.211.124:46023: Network error: Client connection negotiation failed: client connection to 127.18.211.124:46023: connect: Connection refused (error 111)
I20260928 06:44:21.138301 9500 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37627 } attempt: 3
Sep 28 06:44:21 dist-test-slave-b28r krb5kdc[9419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577860, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for kudu/127.18.211.125@KRBTEST.COM
I20260928 06:44:21.145877 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:21.138648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.125:37627 (local address 127.0.0.1:46948)
0928 06:44:21.138724 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:21.138736 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 06:44:21.138827 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:44:21.138956 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:44:21.138957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:44:21.139151 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:44:21.139221 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:44:21.139223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.140167 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.140169 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:44:21.140710 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:44:21.140712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.141201 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.141202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:44:21.141285 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:21.141764 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:44:21.141769 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:44:21.143348 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:44:21.144997 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:44:21.144998 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:44:21.144998 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:44:21.145167 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:44:21.145376 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:44:21.145377 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:44:21.145377 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:44:21.145428 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:44:21.145649 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:44:21.145650 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:44:21.145687 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 06:44:21.145732 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 06:44:21.145810 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 06:44:21.145921 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:21.138724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46948 (local address 127.18.211.125:37627)
0928 06:44:21.138773 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:21.138775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:21.138810 (+ 35us) server_negotiation.cc:407] Connection header received
0928 06:44:21.138875 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:21.138876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:21.138888 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:21.138946 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:21.139291 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.140065 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.140805 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.141127 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.141196 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:21.143448 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:21.143449 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:21.143450 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:21.143458 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:21.144914 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:21.145258 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:21.145259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:21.145260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:21.145322 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:21.145474 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:21.145475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:21.145475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:21.145602 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:21.145661 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:21.145755 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 06:44:21.145831 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:44:21.146364 9500 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.211.124" port: 46023 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:44:21.146670 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:21.146580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:46023 (local address 127.0.0.1:46108)
0928 06:44:21.146624 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:21.146650 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.124:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 06:44:21.146983 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:21.146908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:46023 (local address 127.0.0.1:46122)
0928 06:44:21.146945 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:21.146968 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.124:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260928 06:44:21.147100 9500 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.211.124:46023: Network error: Client connection negotiation failed: client connection to 127.18.211.124:46023: connect: Connection refused (error 111)
W20260928 06:44:21.173952 9582 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 06:44:21.174172 9582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 06:44:21.174188 9582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 06:44:21.174217 9582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 06:44:21.175439 9582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 06:44:21.175491 9582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 06:44:21.175505 9582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 06:44:21.175516 9582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 06:44:21.175530 9582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 06:44:21.177037 9582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.211.84:41339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.211.126:34659,127.18.211.125:37627,127.18.211.124:46023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/krb5kdc/kudu_127.18.211.124.keytab
--principal=kudu/127.18.211.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.211.124:46023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.211.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 28 Sep 2026 05:48:21 UTC on ccycloud.quasar-bvchzj.root.comops.site
build id 4676668
I20260928 06:44:21.177276 9582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 06:44:21.177498 9582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 06:44:21.178246 9500 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.211.124" port: 46023 } attempt: 1
I20260928 06:44:21.178742 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:21.178581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:46023 (local address 127.0.0.1:46138)
0928 06:44:21.178656 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:21.178706 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.124:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 06:44:21.179119 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:21.179004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:46023 (local address 127.0.0.1:46150)
0928 06:44:21.179044 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:21.179073 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.124:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 06:44:21.179284 9500 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.211.124:46023: Network error: Client connection negotiation failed: client connection to 127.18.211.124:46023: connect: Connection refused (error 111)
W20260928 06:44:21.182245 9590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:44:21.182294 9588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 06:44:21.182251 9587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 06:44:21.182597 9582 server_base.cc:1034] running on GCE node
I20260928 06:44:21.182809 9582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 06:44:21.183120 9582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 06:44:21.184270 9582 hybrid_clock.cc:648] HybridClock initialized: now 1790577861184256 us; error 31 us; skew 500 ppm
Sep 28 06:44:21 dist-test-slave-b28r krb5kdc[9419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577861, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 06:44:21.186755 9582 init.cc:377] Logged in from keytab as kudu/127.18.211.124@KRBTEST.COM (short username kudu)
I20260928 06:44:21.187215 9582 webserver.cc:460] Webserver started at http://127.18.211.124:44775/ using document root <none> and password file <none>
I20260928 06:44:21.187444 9582 fs_manager.cc:359] Metadata directory not provided
I20260928 06:44:21.187493 9582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 06:44:21.187619 9582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 06:44:21.188751 9582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-2/data/instance:
uuid: "79d15bbaf16b43638e751ddc52242fa4"
format_stamp: "Formatted at 2026-09-28 06:44:21 on dist-test-slave-b28r"
server_key: "d5856dd2bccd217cdf2840d2effa419c"
server_key_iv: "101ac7ad283561edcab5e528faa9a6f3"
server_key_version: "encryptionkey@0"
I20260928 06:44:21.189090 9582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-2/wal/instance:
uuid: "79d15bbaf16b43638e751ddc52242fa4"
format_stamp: "Formatted at 2026-09-28 06:44:21 on dist-test-slave-b28r"
server_key: "d5856dd2bccd217cdf2840d2effa419c"
server_key_iv: "101ac7ad283561edcab5e528faa9a6f3"
server_key_version: "encryptionkey@0"
I20260928 06:44:21.190366 9582 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260928 06:44:21.191202 9597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 06:44:21.191352 9582 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260928 06:44:21.191440 9582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-2/data,/tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-2/wal
uuid: "79d15bbaf16b43638e751ddc52242fa4"
format_stamp: "Formatted at 2026-09-28 06:44:21 on dist-test-slave-b28r"
server_key: "d5856dd2bccd217cdf2840d2effa419c"
server_key_iv: "101ac7ad283561edcab5e528faa9a6f3"
server_key_version: "encryptionkey@0"
I20260928 06:44:21.191510 9582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 06:44:21.201006 9582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 06:44:21.202793 9582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 06:44:21.202920 9582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 06:44:21.203799 9575 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.211.124" port: 46023 } attempt: 2
I20260928 06:44:21.204205 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:21.204076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:46023 (local address 127.0.0.1:46160)
0928 06:44:21.204135 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:21.204174 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.124:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 06:44:21.204598 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:21.204461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:46023 (local address 127.0.0.1:46166)
0928 06:44:21.204530 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:21.204567 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.211.124:46023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 06:44:21.204701 9575 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.211.124:46023: Network error: Client connection negotiation failed: client connection to 127.18.211.124:46023: connect: Connection refused (error 111)
I20260928 06:44:21.206804 9582 rpc_server.cc:307] RPC server started. Bound to: 127.18.211.124:46023
I20260928 06:44:21.206851 9649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.211.124:46023 every 8 connection(s)
I20260928 06:44:21.207208 9582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-2/data/info.pb
I20260928 06:44:21.208068 9650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 06:44:21.209594 9650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34659 } has no permanent_uuid. Determining permanent_uuid...
I20260928 06:44:21.214560 19277 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskOUYCUS/build/release/bin/kudu as pid 9582
I20260928 06:44:21.214672 19277 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskOUYCUS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790577602522428-19277-0/minicluster-data/master-2/wal/instance
I20260928 06:44:21.214877 19277 external_mini_cluster.cc:1428] Setting key ffaf47f896e70b56f5026af8c5d06bb6
Sep 28 06:44:21 dist-test-slave-b28r krb5kdc[9419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577861, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.124@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
I20260928 06:44:21.218351 9503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:21.210345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33774 (local address 127.18.211.126:34659)
0928 06:44:21.210421 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:21.210422 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:21.210564 (+ 142us) server_negotiation.cc:407] Connection header received
0928 06:44:21.211053 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:21.211054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:21.211064 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:21.211163 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:21.211782 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.212248 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.213226 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.213429 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.213483 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:21.215680 (+ 2197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:21.215681 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:21.215682 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:21.215689 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:21.217045 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:21.217507 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:21.217508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:21.217509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:21.217572 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:21.217790 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:21.217790 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:21.217791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:21.217921 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:21.217980 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:21.218222 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 06:44:21.218287 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 06:44:21.218593 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:21.210289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.126:34659 (local address 127.0.0.1:33774)
0928 06:44:21.210443 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:21.210486 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 06:44:21.210945 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:44:21.211228 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:44:21.211232 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:44:21.211466 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:44:21.211658 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:44:21.211666 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.212353 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.212355 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:44:21.213151 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:44:21.213153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.213493 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.213493 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:44:21.213591 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:21.214038 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:44:21.214046 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:44:21.215577 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:44:21.217119 (+ 1542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:44:21.217126 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:44:21.217128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:44:21.217446 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:44:21.217637 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:44:21.217638 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:44:21.217638 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:44:21.217738 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:44:21.217979 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:44:21.217982 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:44:21.218087 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 06:44:21.218191 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 06:44:21.218382 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20260928 06:44:21.219283 9650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37627 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:44:21 dist-test-slave-b28r krb5kdc[9419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.124@KRBTEST.COM
Sep 28 06:44:21 dist-test-slave-b28r krb5kdc[9419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577861, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.124@KRBTEST.COM for kudu/127.18.211.125@KRBTEST.COM
I20260928 06:44:21.223583 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:21.215367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46172 (local address 127.18.211.124:46023)
0928 06:44:21.215535 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:21.215540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:21.215561 (+ 21us) server_negotiation.cc:407] Connection header received
0928 06:44:21.215580 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:21.215582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:21.215596 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:21.215673 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:21.216431 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.217218 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.218134 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.218561 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.218626 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:21.220907 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:21.220915 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:21.220915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:21.220921 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:21.222392 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:21.222693 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:21.222696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:21.222697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:21.222754 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:21.222989 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:21.222990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:21.222991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:21.223199 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:21.223381 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:21.223429 (+ 48us) server_negotiation.cc:299] Negotiation successful
0928 06:44:21.223496 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":87,"threads_started":1}
I20260928 06:44:21.225608 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:21.219535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.125:37627 (local address 127.0.0.1:46954)
0928 06:44:21.219575 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:21.219594 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 06:44:21.219633 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:44:21.219793 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:44:21.219794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:44:21.219901 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:44:21.219946 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:44:21.219947 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.220558 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.220559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:44:21.221043 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:44:21.221045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.221393 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.221394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:44:21.221458 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:21.221904 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:44:21.221908 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:44:21.223015 (+ 1107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:44:21.224759 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:44:21.224760 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:44:21.224766 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:44:21.224984 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:44:21.225153 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:44:21.225154 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:44:21.225154 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:44:21.225200 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:44:21.225418 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:44:21.225419 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:44:21.225449 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 06:44:21.225490 (+ 41us) client_negotiation.cc:240] Negotiation successful
0928 06:44:21.225550 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260928 06:44:21.225677 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:21.219633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46954 (local address 127.18.211.125:37627)
0928 06:44:21.219680 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:21.219682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:21.219690 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:44:21.219716 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:21.219717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:21.219727 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:21.219792 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:21.220012 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.220483 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.221114 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.221336 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.221390 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:21.223117 (+ 1727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:21.223118 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:21.223118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:21.223125 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:21.224692 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:21.225038 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:21.225039 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:21.225039 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:21.225097 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:21.225247 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:21.225248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:21.225248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:21.225366 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:21.225427 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:21.225521 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 06:44:21.225587 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260928 06:44:21.225978 9650 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.211.124" port: 46023 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 06:44:21 dist-test-slave-b28r krb5kdc[9419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.126@KRBTEST.COM
Sep 28 06:44:21 dist-test-slave-b28r krb5kdc[9419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577861, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.124@KRBTEST.COM for kudu/127.18.211.124@KRBTEST.COM
I20260928 06:44:21.231377 9503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:21.224333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33778 (local address 127.18.211.126:34659)
0928 06:44:21.224381 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:21.224383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:21.224392 (+ 9us) server_negotiation.cc:407] Connection header received
0928 06:44:21.224431 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:21.224432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:21.224442 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:21.224493 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:21.224934 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.225581 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.226607 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.226787 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.226832 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:21.228883 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:21.228884 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:21.228885 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:21.228891 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:21.230305 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:21.230600 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:21.230601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:21.230601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:21.230652 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:21.230835 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:21.230836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:21.230836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:21.230965 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:21.231169 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:21.231220 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 06:44:21.231288 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260928 06:44:21.232303 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:21.226185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:46023 (local address 127.0.0.1:46178)
0928 06:44:21.226224 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:21.226234 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:44:21.226266 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:44:21.226382 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:44:21.226382 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:44:21.226479 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:44:21.226505 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:44:21.226505 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.227395 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.227396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:44:21.227845 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:44:21.227847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.228310 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.228311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:44:21.228354 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:21.228720 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:44:21.228724 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:44:21.229925 (+ 1201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:44:21.231383 (+ 1458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:44:21.231384 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:44:21.231385 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:44:21.231571 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:44:21.231744 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:44:21.231745 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:44:21.231745 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:44:21.231808 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:44:21.232029 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:44:21.232030 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:44:21.232071 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 06:44:21.232134 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 06:44:21.232219 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 06:44:21.232686 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:21.226232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46178 (local address 127.18.211.124:46023)
0928 06:44:21.226263 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:21.226265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:21.226272 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:44:21.226320 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:21.226321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:21.226330 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:21.226378 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:21.226555 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.227128 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.227927 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.228217 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.228287 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:21.230000 (+ 1713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:21.230002 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:21.230002 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:21.230008 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:21.231308 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:21.231634 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:21.231635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:21.231636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:21.231685 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:21.231865 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:21.231865 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:21.231866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:21.231973 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:21.232030 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:21.232156 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 06:44:21.232279 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":16}
I20260928 06:44:21.234421 9650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79d15bbaf16b43638e751ddc52242fa4: Bootstrap starting.
I20260928 06:44:21.235307 9650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79d15bbaf16b43638e751ddc52242fa4: Neither blocks nor log segments found. Creating new log.
I20260928 06:44:21.235566 9650 log.cc:824] T 00000000000000000000000000000000 P 79d15bbaf16b43638e751ddc52242fa4: Log is configured to *not* fsync() on all Append() calls
Sep 28 06:44:21 dist-test-slave-b28r krb5kdc[9419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.211.125@KRBTEST.COM
I20260928 06:44:21.236280 9650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79d15bbaf16b43638e751ddc52242fa4: No bootstrap required, opened a new log
I20260928 06:44:21.237996 9650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79d15bbaf16b43638e751ddc52242fa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446547d6cc9142409a2a7fed33e2d931" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34659 } } peers { permanent_uuid: "b69ba13be608409d84d54f0b75970732" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37627 } } peers { permanent_uuid: "79d15bbaf16b43638e751ddc52242fa4" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 46023 } }
I20260928 06:44:21.238161 9650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79d15bbaf16b43638e751ddc52242fa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79d15bbaf16b43638e751ddc52242fa4, State: Initialized, Role: FOLLOWER
I20260928 06:44:21.238271 9650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79d15bbaf16b43638e751ddc52242fa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446547d6cc9142409a2a7fed33e2d931" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34659 } } peers { permanent_uuid: "b69ba13be608409d84d54f0b75970732" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37627 } } peers { permanent_uuid: "79d15bbaf16b43638e751ddc52242fa4" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 46023 } }
I20260928 06:44:21.238591 9650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79d15bbaf16b43638e751ddc52242fa4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:44:21.238574 9656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79d15bbaf16b43638e751ddc52242fa4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446547d6cc9142409a2a7fed33e2d931" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34659 } } peers { permanent_uuid: "b69ba13be608409d84d54f0b75970732" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37627 } } peers { permanent_uuid: "79d15bbaf16b43638e751ddc52242fa4" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 46023 } } }
I20260928 06:44:21.238682 9656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79d15bbaf16b43638e751ddc52242fa4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:44:21.239147 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:21.232322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46956 (local address 127.18.211.125:37627)
0928 06:44:21.232364 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:21.232366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:21.232441 (+ 75us) server_negotiation.cc:407] Connection header received
0928 06:44:21.232532 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:21.232533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:21.232542 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:21.232592 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:21.233024 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.233504 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.234154 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.234402 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.234458 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:21.236412 (+ 1954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:21.236413 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:21.236413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:21.236420 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:21.237697 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:21.238040 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:21.238042 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:21.238042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:21.238103 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:21.238457 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:21.238459 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:21.238459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:21.238603 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:21.238903 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:21.238972 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 06:44:21.239030 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 06:44:21.240389 9667 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 79d15bbaf16b43638e751ddc52242fa4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:44:21.240486 9667 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:44:21.241007 19277 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 06:44:21.241918 9474 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "446547d6cc9142409a2a7fed33e2d931"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33778
I20260928 06:44:21.240829 9582 master_runner.cc:186] Error getting master registration for 127.18.211.126:34659: OK, instance_id { permanent_uuid: "446547d6cc9142409a2a7fed33e2d931" instance_seqno: 1790577860913341 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:44:21.243009 9582 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:44:21.254415 9500 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.211.124" port: 46023 } attempt: 2
Sep 28 06:44:21 dist-test-slave-b28r krb5kdc[9419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577860, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.126@KRBTEST.COM for kudu/127.18.211.124@KRBTEST.COM
I20260928 06:44:21.261073 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:21.254733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:46023 (local address 127.0.0.1:46194)
0928 06:44:21.254787 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:21.254798 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 06:44:21.254844 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:44:21.255096 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:44:21.255097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:44:21.255264 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:44:21.255306 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:44:21.255308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.255914 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.255915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:44:21.256435 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:44:21.256437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.256766 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.256766 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:44:21.256828 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:21.257228 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:44:21.257233 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:44:21.258359 (+ 1126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:44:21.260076 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:44:21.260077 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:44:21.260078 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:44:21.260250 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:44:21.260417 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:44:21.260417 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:44:21.260418 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:44:21.260489 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:44:21.260766 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:44:21.260767 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:44:21.260813 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 06:44:21.260872 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 06:44:21.260975 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 06:44:21.261073 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:21.254922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46194 (local address 127.18.211.124:46023)
0928 06:44:21.254977 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:21.254978 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:21.254986 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:44:21.255009 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:21.255010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:21.255019 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:21.255061 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:21.255377 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.255847 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.256508 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.256710 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.256763 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:21.258441 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:21.258442 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:21.258443 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:21.258450 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:21.260013 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:21.260312 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:21.260313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:21.260314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:21.260370 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:21.260550 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:21.260551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:21.260552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:21.260697 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:21.260761 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:21.260899 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 06:44:21.260975 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:44:21.262410 9500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 446547d6cc9142409a2a7fed33e2d931: Bootstrap starting.
I20260928 06:44:21.263034 9500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 446547d6cc9142409a2a7fed33e2d931: Neither blocks nor log segments found. Creating new log.
I20260928 06:44:21.263329 9500 log.cc:824] T 00000000000000000000000000000000 P 446547d6cc9142409a2a7fed33e2d931: Log is configured to *not* fsync() on all Append() calls
I20260928 06:44:21.264027 9500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 446547d6cc9142409a2a7fed33e2d931: No bootstrap required, opened a new log
I20260928 06:44:21.265002 9500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 446547d6cc9142409a2a7fed33e2d931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446547d6cc9142409a2a7fed33e2d931" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34659 } } peers { permanent_uuid: "b69ba13be608409d84d54f0b75970732" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37627 } } peers { permanent_uuid: "79d15bbaf16b43638e751ddc52242fa4" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 46023 } }
I20260928 06:44:21.265111 9500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 446547d6cc9142409a2a7fed33e2d931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446547d6cc9142409a2a7fed33e2d931, State: Initialized, Role: FOLLOWER
I20260928 06:44:21.265183 9500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 446547d6cc9142409a2a7fed33e2d931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446547d6cc9142409a2a7fed33e2d931" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34659 } } peers { permanent_uuid: "b69ba13be608409d84d54f0b75970732" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37627 } } peers { permanent_uuid: "79d15bbaf16b43638e751ddc52242fa4" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 46023 } }
I20260928 06:44:21.265436 9670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 446547d6cc9142409a2a7fed33e2d931 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446547d6cc9142409a2a7fed33e2d931" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34659 } } peers { permanent_uuid: "b69ba13be608409d84d54f0b75970732" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37627 } } peers { permanent_uuid: "79d15bbaf16b43638e751ddc52242fa4" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 46023 } } }
I20260928 06:44:21.265489 9500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 446547d6cc9142409a2a7fed33e2d931 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 06:44:21.265527 9670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 446547d6cc9142409a2a7fed33e2d931 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:44:21.265626 9474 raft_consensus.cc:482] T 00000000000000000000000000000000 P 446547d6cc9142409a2a7fed33e2d931 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 06:44:21.265744 9474 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 446547d6cc9142409a2a7fed33e2d931 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:44:21.266566 9474 raft_consensus.cc:504] T 00000000000000000000000000000000 P 446547d6cc9142409a2a7fed33e2d931 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446547d6cc9142409a2a7fed33e2d931" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34659 } } peers { permanent_uuid: "b69ba13be608409d84d54f0b75970732" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37627 } } peers { permanent_uuid: "79d15bbaf16b43638e751ddc52242fa4" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 46023 } }
W20260928 06:44:21.267342 9681 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 446547d6cc9142409a2a7fed33e2d931: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:44:21.267395 9681 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:44:21.268066 9624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "446547d6cc9142409a2a7fed33e2d931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "79d15bbaf16b43638e751ddc52242fa4"
I20260928 06:44:21.268190 9624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79d15bbaf16b43638e751ddc52242fa4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:44:21.269055 9624 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 79d15bbaf16b43638e751ddc52242fa4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 446547d6cc9142409a2a7fed33e2d931 in term 1.
I20260928 06:44:21.269325 9448 leader_election.cc:302] T 00000000000000000000000000000000 P 446547d6cc9142409a2a7fed33e2d931 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 446547d6cc9142409a2a7fed33e2d931, 79d15bbaf16b43638e751ddc52242fa4; no voters:
I20260928 06:44:21.269449 9670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 446547d6cc9142409a2a7fed33e2d931 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 06:44:21.269564 9670 raft_consensus.cc:686] T 00000000000000000000000000000000 P 446547d6cc9142409a2a7fed33e2d931 [term 1 LEADER]: Becoming Leader. State: Replica: 446547d6cc9142409a2a7fed33e2d931, State: Running, Role: LEADER
I20260928 06:44:21.269639 9670 consensus_queue.cc:237] T 00000000000000000000000000000000 P 446547d6cc9142409a2a7fed33e2d931 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446547d6cc9142409a2a7fed33e2d931" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34659 } } peers { permanent_uuid: "b69ba13be608409d84d54f0b75970732" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37627 } } peers { permanent_uuid: "79d15bbaf16b43638e751ddc52242fa4" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 46023 } }
I20260928 06:44:21.270025 9670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 446547d6cc9142409a2a7fed33e2d931 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 446547d6cc9142409a2a7fed33e2d931. Latest consensus state: current_term: 1 leader_uuid: "446547d6cc9142409a2a7fed33e2d931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446547d6cc9142409a2a7fed33e2d931" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34659 } } peers { permanent_uuid: "b69ba13be608409d84d54f0b75970732" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37627 } } peers { permanent_uuid: "79d15bbaf16b43638e751ddc52242fa4" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 46023 } } }
I20260928 06:44:21.270099 9670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 446547d6cc9142409a2a7fed33e2d931 [sys.catalog]: This master's current role is: LEADER
I20260928 06:44:21.270427 9474 leader_election.cc:288] T 00000000000000000000000000000000 P 446547d6cc9142409a2a7fed33e2d931 [CANDIDATE]: Term 1 election: Requested vote from peers b69ba13be608409d84d54f0b75970732 (127.18.211.125:37627), 79d15bbaf16b43638e751ddc52242fa4 (127.18.211.124:46023)
I20260928 06:44:21.268512 9432 master_runner.cc:186] Error getting master registration for 127.18.211.125:37627: OK, instance_id { permanent_uuid: "b69ba13be608409d84d54f0b75970732" instance_seqno: 1790577861061669 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 06:44:21.270671 9432 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 06:44:21.272490 9685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 06:44:21.272857 9685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 06:44:21.279439 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:21.271761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46202 (local address 127.18.211.124:46023)
0928 06:44:21.271807 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:21.271809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:21.271925 (+ 116us) server_negotiation.cc:407] Connection header received
0928 06:44:21.272020 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:21.272021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:21.272031 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:21.272060 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:21.272890 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.273614 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.274654 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.274947 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.274989 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:21.276571 (+ 1582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:21.276573 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:21.276573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:21.276580 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:21.278196 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:21.278589 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:21.278590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:21.278591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:21.278644 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:21.278832 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:21.278833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:21.278833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:21.278964 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:21.279032 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:21.279281 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 06:44:21.279356 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260928 06:44:21.286055 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:21.272134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46964 (local address 127.18.211.125:37627)
0928 06:44:21.272171 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:21.272173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:21.275451 (+ 3278us) server_negotiation.cc:407] Connection header received
0928 06:44:21.275550 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:21.275551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:21.275561 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:21.275590 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:21.279993 (+ 4403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.280718 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.281713 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.282034 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.282072 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:21.283402 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:21.283403 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:21.283404 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:21.283411 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:21.284874 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:21.285259 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:21.285260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:21.285261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:21.285314 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:21.285493 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:21.285493 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:21.285494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:21.285626 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:21.285701 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:21.285882 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 06:44:21.285961 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260928 06:44:21.287370 9503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:21.271624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33792 (local address 127.18.211.126:34659)
0928 06:44:21.271661 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:21.271663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:21.274107 (+ 2444us) server_negotiation.cc:407] Connection header received
0928 06:44:21.274244 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:21.274245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:21.274267 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:21.274296 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:21.275635 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.276351 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.278389 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.278698 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.278736 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:21.281632 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:21.281645 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:21.281646 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:21.281652 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:21.283915 (+ 2263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:21.285253 (+ 1338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:21.285254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:21.285255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:21.285327 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:21.285892 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:21.285893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:21.285894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:21.286040 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:21.286123 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:21.287191 (+ 1068us) server_negotiation.cc:299] Negotiation successful
0928 06:44:21.287278 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:44:21.287941 9624 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 79d15bbaf16b43638e751ddc52242fa4 [term 1 FOLLOWER]: Refusing update from remote peer 446547d6cc9142409a2a7fed33e2d931: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:44:21.288353 9683 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 446547d6cc9142409a2a7fed33e2d931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79d15bbaf16b43638e751ddc52242fa4" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 46023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:44:21.289753 9656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79d15bbaf16b43638e751ddc52242fa4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 446547d6cc9142409a2a7fed33e2d931. Latest consensus state: current_term: 1 leader_uuid: "446547d6cc9142409a2a7fed33e2d931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446547d6cc9142409a2a7fed33e2d931" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34659 } } peers { permanent_uuid: "b69ba13be608409d84d54f0b75970732" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37627 } } peers { permanent_uuid: "79d15bbaf16b43638e751ddc52242fa4" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 46023 } } }
I20260928 06:44:21.289851 9656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79d15bbaf16b43638e751ddc52242fa4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:44:21.291097 9670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 446547d6cc9142409a2a7fed33e2d931 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "446547d6cc9142409a2a7fed33e2d931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446547d6cc9142409a2a7fed33e2d931" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34659 } } peers { permanent_uuid: "b69ba13be608409d84d54f0b75970732" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37627 } } peers { permanent_uuid: "79d15bbaf16b43638e751ddc52242fa4" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 46023 } } }
I20260928 06:44:21.291184 9670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 446547d6cc9142409a2a7fed33e2d931 [sys.catalog]: This master's current role is: LEADER
I20260928 06:44:21.291332 9656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79d15bbaf16b43638e751ddc52242fa4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "446547d6cc9142409a2a7fed33e2d931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446547d6cc9142409a2a7fed33e2d931" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34659 } } peers { permanent_uuid: "b69ba13be608409d84d54f0b75970732" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37627 } } peers { permanent_uuid: "79d15bbaf16b43638e751ddc52242fa4" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 46023 } } }
I20260928 06:44:21.291407 9656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79d15bbaf16b43638e751ddc52242fa4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:44:21.291787 9693 mvcc.cc:205] Tried to move back new op lower bound from 7334206919825588224 to 7334206919761108992. Current Snapshot: MvccSnapshot[applied={T|T < 7334206919825588224}]
I20260928 06:44:21.291949 9694 mvcc.cc:205] Tried to move back new op lower bound from 7334206919825588224 to 7334206919761108992. Current Snapshot: MvccSnapshot[applied={T|T < 7334206919825588224 or (T in {7334206919825588224})}]
I20260928 06:44:21.292412 9685 catalog_manager.cc:1347] Generated new cluster ID: 71cc159e75704214b3e776940320d0ce
I20260928 06:44:21.292484 9685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 06:44:21.295811 9575 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.211.124" port: 46023 } attempt: 3
Sep 28 06:44:21 dist-test-slave-b28r krb5kdc[9419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790577861, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.211.125@KRBTEST.COM for kudu/127.18.211.124@KRBTEST.COM
I20260928 06:44:21.306864 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:21.296070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.211.124:46023 (local address 127.0.0.1:46214)
0928 06:44:21.296119 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0928 06:44:21.296129 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0928 06:44:21.296166 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 06:44:21.299441 (+ 3275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 06:44:21.299442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 06:44:21.299634 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 06:44:21.299685 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:44:21.299687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.300595 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.300596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:44:21.301413 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 06:44:21.301415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.301867 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.301868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 06:44:21.301924 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:21.302398 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 06:44:21.302404 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 06:44:21.303853 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 06:44:21.305807 (+ 1954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:44:21.305808 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:44:21.305809 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:44:21.306006 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:44:21.306208 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 06:44:21.306209 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 06:44:21.306209 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0928 06:44:21.306267 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 06:44:21.306540 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 06:44:21.306541 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 06:44:21.306589 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 06:44:21.306648 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 06:44:21.306757 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 06:44:21.307232 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:21.299251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46214 (local address 127.18.211.124:46023)
0928 06:44:21.299321 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:21.299324 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:21.299334 (+ 10us) server_negotiation.cc:407] Connection header received
0928 06:44:21.299358 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:21.299359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:21.299370 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:21.299431 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:21.299755 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.300516 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.301487 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:21.301796 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:21.301868 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:21.303931 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 06:44:21.303933 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 06:44:21.303933 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 06:44:21.303941 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 06:44:21.305739 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:21.306082 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:21.306083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:21.306083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:21.306143 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 06:44:21.306329 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 06:44:21.306330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 06:44:21.306331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 06:44:21.306459 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 06:44:21.306515 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:21.307049 (+ 534us) server_negotiation.cc:299] Negotiation successful
0928 06:44:21.307145 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:44:21.308911 9575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b69ba13be608409d84d54f0b75970732: Bootstrap starting.
I20260928 06:44:21.309650 9575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b69ba13be608409d84d54f0b75970732: Neither blocks nor log segments found. Creating new log.
I20260928 06:44:21.309978 9575 log.cc:824] T 00000000000000000000000000000000 P b69ba13be608409d84d54f0b75970732: Log is configured to *not* fsync() on all Append() calls
I20260928 06:44:21.315774 9575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b69ba13be608409d84d54f0b75970732: No bootstrap required, opened a new log
I20260928 06:44:21.318523 9575 raft_consensus.cc:348] T 00000000000000000000000000000000 P b69ba13be608409d84d54f0b75970732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446547d6cc9142409a2a7fed33e2d931" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34659 } } peers { permanent_uuid: "b69ba13be608409d84d54f0b75970732" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37627 } } peers { permanent_uuid: "79d15bbaf16b43638e751ddc52242fa4" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 46023 } }
I20260928 06:44:21.318764 9575 raft_consensus.cc:729] T 00000000000000000000000000000000 P b69ba13be608409d84d54f0b75970732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b69ba13be608409d84d54f0b75970732, State: Initialized, Role: FOLLOWER
I20260928 06:44:21.318986 9575 consensus_queue.cc:260] T 00000000000000000000000000000000 P b69ba13be608409d84d54f0b75970732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446547d6cc9142409a2a7fed33e2d931" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34659 } } peers { permanent_uuid: "b69ba13be608409d84d54f0b75970732" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37627 } } peers { permanent_uuid: "79d15bbaf16b43638e751ddc52242fa4" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 46023 } }
I20260928 06:44:21.319716 9549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "446547d6cc9142409a2a7fed33e2d931" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b69ba13be608409d84d54f0b75970732"
I20260928 06:44:21.319818 9549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b69ba13be608409d84d54f0b75970732 [term 0 FOLLOWER]: Advancing to term 1
I20260928 06:44:21.320647 9549 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b69ba13be608409d84d54f0b75970732 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 446547d6cc9142409a2a7fed33e2d931 in term 1.
I20260928 06:44:21.321074 9548 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b69ba13be608409d84d54f0b75970732 [term 1 FOLLOWER]: Refusing update from remote peer 446547d6cc9142409a2a7fed33e2d931: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 06:44:21.321347 9683 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 446547d6cc9142409a2a7fed33e2d931 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b69ba13be608409d84d54f0b75970732" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 06:44:21.322779 9575 sys_catalog.cc:564] T 00000000000000000000000000000000 P b69ba13be608409d84d54f0b75970732 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 06:44:21.325510 9714 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b69ba13be608409d84d54f0b75970732: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 06:44:21.325606 9714 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 06:44:21.325877 9703 sys_catalog.cc:455] T 00000000000000000000000000000000 P b69ba13be608409d84d54f0b75970732 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "446547d6cc9142409a2a7fed33e2d931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446547d6cc9142409a2a7fed33e2d931" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34659 } } peers { permanent_uuid: "b69ba13be608409d84d54f0b75970732" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37627 } } peers { permanent_uuid: "79d15bbaf16b43638e751ddc52242fa4" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 46023 } } }
I20260928 06:44:21.326000 9703 sys_catalog.cc:458] T 00000000000000000000000000000000 P b69ba13be608409d84d54f0b75970732 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:44:21.326083 9703 sys_catalog.cc:455] T 00000000000000000000000000000000 P b69ba13be608409d84d54f0b75970732 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 446547d6cc9142409a2a7fed33e2d931. Latest consensus state: current_term: 1 leader_uuid: "446547d6cc9142409a2a7fed33e2d931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446547d6cc9142409a2a7fed33e2d931" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34659 } } peers { permanent_uuid: "b69ba13be608409d84d54f0b75970732" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37627 } } peers { permanent_uuid: "79d15bbaf16b43638e751ddc52242fa4" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 46023 } } }
I20260928 06:44:21.326176 9703 sys_catalog.cc:458] T 00000000000000000000000000000000 P b69ba13be608409d84d54f0b75970732 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:44:21.326253 9703 sys_catalog.cc:455] T 00000000000000000000000000000000 P b69ba13be608409d84d54f0b75970732 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "446547d6cc9142409a2a7fed33e2d931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446547d6cc9142409a2a7fed33e2d931" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34659 } } peers { permanent_uuid: "b69ba13be608409d84d54f0b75970732" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37627 } } peers { permanent_uuid: "79d15bbaf16b43638e751ddc52242fa4" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 46023 } } }
I20260928 06:44:21.326337 9703 sys_catalog.cc:458] T 00000000000000000000000000000000 P b69ba13be608409d84d54f0b75970732 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 06:44:21.327574 9701 mvcc.cc:205] Tried to move back new op lower bound from 7334206919825588224 to 7334206919761108992. Current Snapshot: MvccSnapshot[applied={T|T < 7334206919825588224}]
I20260928 06:44:21.337756 9683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 446547d6cc9142409a2a7fed33e2d931 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "446547d6cc9142409a2a7fed33e2d931" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "446547d6cc9142409a2a7fed33e2d931" member_type: VOTER last_known_addr { host: "127.18.211.126" port: 34659 } } peers { permanent_uuid: "b69ba13be608409d84d54f0b75970732" member_type: VOTER last_known_addr { host: "127.18.211.125" port: 37627 } } peers { permanent_uuid: "79d15bbaf16b43638e751ddc52242fa4" member_type: VOTER last_known_addr { host: "127.18.211.124" port: 46023 } } }
I20260928 06:44:21.337899 9683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 446547d6cc9142409a2a7fed33e2d931 [sys.catalog]: This master's current role is: LEADER
I20260928 06:44:21.339377 9685 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 06:44:21.340175 9685 catalog_manager.cc:1504] Loading token signing keys...
I20260928 06:44:21.349432 9685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 446547d6cc9142409a2a7fed33e2d931: Generated new TSK 0
I20260928 06:44:21.349625 9685 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 06:44:21.368677 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:44:21.367273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46980 (local address 127.18.211.125:37627)
0928 06:44:21.367316 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:21.367319 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:21.367983 (+ 664us) server_negotiation.cc:407] Connection header received
0928 06:44:21.368086 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:21.368087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:21.368098 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:21.368127 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:21.368605 (+ 478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46980: BlockingRecv error: recv got EOF from 127.0.0.1:46980 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 06:44:21.369601 9503 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:44:21.367688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33806 (local address 127.18.211.126:34659)
0928 06:44:21.367728 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:21.367730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:21.369080 (+ 1350us) server_negotiation.cc:407] Connection header received
0928 06:44:21.369172 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:21.369173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:21.369183 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:21.369224 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:21.369535 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33806: BlockingRecv error: recv got EOF from 127.0.0.1:33806 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 06:44:21.370805 9654 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:44:21.370066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46222 (local address 127.18.211.124:46023)
0928 06:44:21.370107 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:21.370108 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:21.370308 (+ 200us) server_negotiation.cc:407] Connection header received
0928 06:44:21.370405 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:21.370406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:21.370416 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:21.370445 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:21.370748 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46222: BlockingRecv error: recv got EOF from 127.0.0.1:46222 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260928 06:44:21.373167 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:44:21.372379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46982 (local address 127.18.211.125:37627)
0928 06:44:21.372413 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:21.372415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:21.372648 (+ 233us) server_negotiation.cc:407] Connection header received
0928 06:44:21.372756 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:21.372757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:21.372776 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:21.372805 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:21.373121 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46982: BlockingRecv error: recv got EOF from 127.0.0.1:46982 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260928 06:44:21.376295 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:44:21.375495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46992 (local address 127.18.211.125:37627)
0928 06:44:21.375539 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:21.375541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:21.375794 (+ 253us) server_negotiation.cc:407] Connection header received
0928 06:44:21.375887 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:21.375889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:21.375899 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:21.375926 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:21.376250 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46992: BlockingRecv error: recv got EOF from 127.0.0.1:46992 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260928 06:44:21.380409 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:44:21.379622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46998 (local address 127.18.211.125:37627)
0928 06:44:21.379656 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:21.379657 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:21.379909 (+ 252us) server_negotiation.cc:407] Connection header received
0928 06:44:21.380004 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:21.380005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:21.380015 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:21.380043 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:21.380368 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46998: BlockingRecv error: recv got EOF from 127.0.0.1:46998 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260928 06:44:21.386322 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:44:21.385534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47000 (local address 127.18.211.125:37627)
0928 06:44:21.385568 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:21.385570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:21.385813 (+ 243us) server_negotiation.cc:407] Connection header received
0928 06:44:21.385907 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:21.385907 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:21.385918 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:21.385945 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:21.386279 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47000: BlockingRecv error: recv got EOF from 127.0.0.1:47000 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260928 06:44:21.396595 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:44:21.395690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47002 (local address 127.18.211.125:37627)
0928 06:44:21.395731 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:21.395733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:21.396002 (+ 269us) server_negotiation.cc:407] Connection header received
0928 06:44:21.396103 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:21.396104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:21.396116 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:21.396146 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:21.396526 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47002: BlockingRecv error: recv got EOF from 127.0.0.1:47002 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260928 06:44:21.414774 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:44:21.414072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47008 (local address 127.18.211.125:37627)
0928 06:44:21.414115 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:21.414116 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:21.414239 (+ 123us) server_negotiation.cc:407] Connection header received
0928 06:44:21.414292 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:21.414293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:21.414303 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:21.414369 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:21.414714 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47008: BlockingRecv error: recv got EOF from 127.0.0.1:47008 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260928 06:44:21.448743 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:44:21.448009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47012 (local address 127.18.211.125:37627)
0928 06:44:21.448074 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:21.448076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:21.448151 (+ 75us) server_negotiation.cc:407] Connection header received
0928 06:44:21.448201 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:21.448203 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:21.448216 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:21.448302 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:21.448680 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47012: BlockingRecv error: recv got EOF from 127.0.0.1:47012 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260928 06:44:21.514575 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:44:21.513860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47018 (local address 127.18.211.125:37627)
0928 06:44:21.513933 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:21.513935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:21.514006 (+ 71us) server_negotiation.cc:407] Connection header received
0928 06:44:21.514081 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:21.514082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:21.514093 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:21.514143 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:21.514520 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47018: BlockingRecv error: recv got EOF from 127.0.0.1:47018 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 06:44:21.644685 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:44:21.643854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47032 (local address 127.18.211.125:37627)
0928 06:44:21.643914 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:21.643916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:21.644027 (+ 111us) server_negotiation.cc:407] Connection header received
0928 06:44:21.644116 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:21.644117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:21.644174 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:21.644234 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:21.644617 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47032: BlockingRecv error: recv got EOF from 127.0.0.1:47032 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260928 06:44:21.902419 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 06:44:21.901815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47048 (local address 127.18.211.125:37627)
0928 06:44:21.901878 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:21.901880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:21.901892 (+ 12us) server_negotiation.cc:407] Connection header received
0928 06:44:21.901970 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:21.901971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:21.901981 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:21.902025 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0928 06:44:21.902356 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47048: BlockingRecv error: recv got EOF from 127.0.0.1:47048 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 06:44:22.240964 9667 catalog_manager.cc:1259] Loaded cluster ID: 71cc159e75704214b3e776940320d0ce
I20260928 06:44:22.241043 9667 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 79d15bbaf16b43638e751ddc52242fa4: loading cluster ID for follower catalog manager: success
I20260928 06:44:22.241907 9667 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 79d15bbaf16b43638e751ddc52242fa4: acquiring CA information for follower catalog manager: success
I20260928 06:44:22.242234 9667 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 79d15bbaf16b43638e751ddc52242fa4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:44:22.326100 9714 catalog_manager.cc:1259] Loaded cluster ID: 71cc159e75704214b3e776940320d0ce
I20260928 06:44:22.326182 9714 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b69ba13be608409d84d54f0b75970732: loading cluster ID for follower catalog manager: success
I20260928 06:44:22.327001 9714 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b69ba13be608409d84d54f0b75970732: acquiring CA information for follower catalog manager: success
I20260928 06:44:22.327296 9714 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b69ba13be608409d84d54f0b75970732: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 06:44:22.418195 9781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:22.415618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47052 (local address 127.18.211.125:37627)
0928 06:44:22.415746 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:22.415748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:22.415779 (+ 31us) server_negotiation.cc:407] Connection header received
0928 06:44:22.415846 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:22.415847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:22.415899 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:22.415971 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:44:22.416149 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:22.416652 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:22.417481 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:22.417673 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:22.417722 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:22.417895 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:44:22.418006 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:44:22.418062 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:22.418099 (+ 37us) server_negotiation.cc:299] Negotiation successful
0928 06:44:22.418115 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":72,"threads_started":1}
I20260928 06:44:22.422258 9787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:22.419914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46232 (local address 127.18.211.124:46023)
0928 06:44:22.420030 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:22.420031 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:22.420038 (+ 7us) server_negotiation.cc:407] Connection header received
0928 06:44:22.420080 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:22.420081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:22.420117 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:22.420185 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:44:22.420324 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:22.420812 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:22.421516 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:22.421721 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:22.421807 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:22.421941 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:44:22.422045 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:44:22.422112 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:22.422150 (+ 38us) server_negotiation.cc:299] Negotiation successful
0928 06:44:22.422173 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":56,"threads_started":1}
I20260928 06:44:22.426491 9787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:22.423976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46234 (local address 127.18.211.124:46023)
0928 06:44:22.424016 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:22.424018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:22.424175 (+ 157us) server_negotiation.cc:407] Connection header received
0928 06:44:22.424226 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:22.424226 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:22.424236 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:22.424293 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:44:22.424396 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:22.424907 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:22.425754 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:22.426052 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:22.426115 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:22.426268 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:44:22.426302 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:44:22.426340 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:22.426400 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 06:44:22.426420 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260928 06:44:22.426854 9781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:22.423860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47062 (local address 127.18.211.125:37627)
0928 06:44:22.423897 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:22.423899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:22.424193 (+ 294us) server_negotiation.cc:407] Connection header received
0928 06:44:22.424235 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:22.424236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:22.424247 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:22.424293 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:44:22.424521 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:22.425274 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:22.426103 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:22.426358 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:22.426413 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:22.426571 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:44:22.426607 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:44:22.426654 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:22.426746 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 06:44:22.426782 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260928 06:44:22.427320 9795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 06:44:22.424004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33814 (local address 127.18.211.126:34659)
0928 06:44:22.424257 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 06:44:22.424258 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0928 06:44:22.424266 (+ 8us) server_negotiation.cc:407] Connection header received
0928 06:44:22.424319 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 06:44:22.424320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 06:44:22.424368 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 06:44:22.424451 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 06:44:22.424561 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:22.425347 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:22.426230 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 06:44:22.426540 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 06:44:22.426680 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 06:44:22.426900 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 06:44:22.427008 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 06:44:22.427062 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 06:44:22.427210 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 06:44:22.427240 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":45,"threads_started":1}
I20260928 06:44:22.428918 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 9432
I20260928 06:44:22.433612 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 9507
I20260928 06:44:22.438818 19277 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskOUYCUS/build/release/bin/kudu with pid 9582
2026-09-28T06:44:22Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2708 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5456 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (259922 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST